[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:11 dist-test-slave-r7hf krb5kdc[1065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:11 dist-test-slave-r7hf krb5kdc[1065](info): set up 2 sockets
Sep 24 17:43:11 dist-test-slave-r7hf krb5kdc[1065](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 17:43:12.444747 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:12 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:12.573988 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:33633
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:33633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:12.658377 1078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:12.658643 1078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:12.658667 1078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:12.659998 1078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:12.660050 1078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:12.660065 1078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:12.660079 1078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:12.660091 1078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:12.661604 1078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:33633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:12.661839 1078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:12.662120 1078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:12.667706 1084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:12.667727 1078 server_base.cc:1034] running on GCE node
W20260924 17:43:12.667871 1086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:12.667711 1083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:12.668305 1078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:12.668654 1078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:12.669895 1078 hybrid_clock.cc:648] HybridClock initialized: now 1790271792669876 us; error 52 us; skew 500 ppm
Sep 24 17:43:12 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271792, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:12.672627 1078 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:12.673247 1078 webserver.cc:460] Webserver started at http://127.1.8.190:36007/ using document root <none> and password file <none>
I20260924 17:43:12.673502 1078 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:12.673584 1078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:12.673741 1078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:12.674960 1078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "ad82222cc4db40c4955c7250be034efc"
format_stamp: "Formatted at 2026-09-24 17:43:12 on dist-test-slave-r7hf"
server_key: "4037cb8353917f523aed255cf94aaecf"
server_key_iv: "7eaa09421f2fc0dd643f9c031d970bab"
server_key_version: "encryptionkey@0"
I20260924 17:43:12.675364 1078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "ad82222cc4db40c4955c7250be034efc"
format_stamp: "Formatted at 2026-09-24 17:43:12 on dist-test-slave-r7hf"
server_key: "4037cb8353917f523aed255cf94aaecf"
server_key_iv: "7eaa09421f2fc0dd643f9c031d970bab"
server_key_version: "encryptionkey@0"
I20260924 17:43:12.676915 1078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:12.677798 1093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:12.678017 1078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:12.678139 1078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "ad82222cc4db40c4955c7250be034efc"
format_stamp: "Formatted at 2026-09-24 17:43:12 on dist-test-slave-r7hf"
server_key: "4037cb8353917f523aed255cf94aaecf"
server_key_iv: "7eaa09421f2fc0dd643f9c031d970bab"
server_key_version: "encryptionkey@0"
I20260924 17:43:12.678228 1078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:12.695374 1078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:12.697716 1078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:12.697935 1078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:12.702618 1078 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:33633
I20260924 17:43:12.702647 1145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:33633 every 8 connection(s)
I20260924 17:43:12.703063 1078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:12.703967 1146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:12.706368 1146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc: Bootstrap starting.
I20260924 17:43:12.707026 1146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:12.707279 1146 log.cc:824] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:12.708024 1146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc: No bootstrap required, opened a new log
I20260924 17:43:12.709420 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1078
I20260924 17:43:12.709530 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:12.710009 1146 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82222cc4db40c4955c7250be034efc" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33633 } }
I20260924 17:43:12.710201 1146 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:12.710240 1146 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad82222cc4db40c4955c7250be034efc, State: Initialized, Role: FOLLOWER
I20260924 17:43:12.710309 1058 external_mini_cluster.cc:1428] Setting key 6a1de1a979bb557810c70f76d36084e5
I20260924 17:43:12.710430 1146 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82222cc4db40c4955c7250be034efc" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33633 } }
I20260924 17:43:12.710527 1146 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:12.710559 1146 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:12.710611 1146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:12.711326 1146 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82222cc4db40c4955c7250be034efc" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33633 } }
I20260924 17:43:12.711463 1146 leader_election.cc:302] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad82222cc4db40c4955c7250be034efc; no voters:
I20260924 17:43:12.711652 1146 leader_election.cc:288] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:12.711683 1151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:12.711930 1146 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:12.711942 1151 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [term 1 LEADER]: Becoming Leader. State: Replica: ad82222cc4db40c4955c7250be034efc, State: Running, Role: LEADER
I20260924 17:43:12.712067 1151 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82222cc4db40c4955c7250be034efc" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33633 } }
I20260924 17:43:12.712541 1153 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad82222cc4db40c4955c7250be034efc. Latest consensus state: current_term: 1 leader_uuid: "ad82222cc4db40c4955c7250be034efc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82222cc4db40c4955c7250be034efc" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33633 } } }
I20260924 17:43:12.712632 1153 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:12.712805 1152 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad82222cc4db40c4955c7250be034efc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad82222cc4db40c4955c7250be034efc" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33633 } } }
I20260924 17:43:12.712886 1152 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc [sys.catalog]: This master's current role is: LEADER
W20260924 17:43:12.713891 1166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:12.713948 1166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:12.713996 1160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:12.714159 1160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:12.715790 1160 catalog_manager.cc:1347] Generated new cluster ID: 3f74d9d576dc43b4840a0c484cf6bbc3
I20260924 17:43:12.715847 1160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:43:12 dist-test-slave-r7hf krb5kdc[1065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:12.727116 1160 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:12.727819 1160 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:12.729070 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:12.711278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.1.8.190:33633)
0924 17:43:12.711575 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:12.711586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:12.713314 (+ 1728us) server_negotiation.cc:407] Connection header received
0924 17:43:12.714356 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:12.714363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:12.714471 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:12.714619 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:12.716516 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:12.717448 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:12.718595 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:12.719221 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:12.719345 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:12.723934 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:12.723940 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:12.723941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:12.723955 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:12.726807 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:12.727431 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:12.727436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:12.727436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:12.727557 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:12.727891 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:12.727892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:12.727893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:12.728180 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:12.728341 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:12.728665 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 17:43:12.728826 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":127,"threads_started":1}
I20260924 17:43:12.731339 1160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad82222cc4db40c4955c7250be034efc: Generated new TSK 0
I20260924 17:43:12.731564 1160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:12.761772 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:33633
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:12.847651 1172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:12.847890 1172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:12.847918 1172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:12.849216 1172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:12.849269 1172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:12.849335 1172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:12.851003 1172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:33633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:12.851280 1172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:12.851547 1172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:12.857041 1178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:12.857036 1180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:12.857041 1177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:12.857491 1172 server_base.cc:1034] running on GCE node
I20260924 17:43:12.857722 1172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:12.858088 1172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:12.859246 1172 hybrid_clock.cc:648] HybridClock initialized: now 1790271792859223 us; error 46 us; skew 500 ppm
Sep 24 17:43:12 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271792, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:12.861927 1172 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:12.862514 1172 webserver.cc:460] Webserver started at http://127.1.8.129:44607/ using document root <none> and password file <none>
I20260924 17:43:12.862752 1172 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:12.862833 1172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:12.862994 1172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:12.864136 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "0635fe139f994bb3808c0a1691e7f1a0"
format_stamp: "Formatted at 2026-09-24 17:43:12 on dist-test-slave-r7hf"
server_key: "b2e8c808814aaab9cf03249b1c452415"
server_key_iv: "cd5fffa782d180d999f17b17a8129a40"
server_key_version: "encryptionkey@0"
I20260924 17:43:12.864549 1172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "0635fe139f994bb3808c0a1691e7f1a0"
format_stamp: "Formatted at 2026-09-24 17:43:12 on dist-test-slave-r7hf"
server_key: "b2e8c808814aaab9cf03249b1c452415"
server_key_iv: "cd5fffa782d180d999f17b17a8129a40"
server_key_version: "encryptionkey@0"
I20260924 17:43:12.866120 1172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:12.867046 1187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:12.867241 1172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:12.867348 1172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "0635fe139f994bb3808c0a1691e7f1a0"
format_stamp: "Formatted at 2026-09-24 17:43:12 on dist-test-slave-r7hf"
server_key: "b2e8c808814aaab9cf03249b1c452415"
server_key_iv: "cd5fffa782d180d999f17b17a8129a40"
server_key_version: "encryptionkey@0"
I20260924 17:43:12.867446 1172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:12.888794 1172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:12.890940 1172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:12.891120 1172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:12.891395 1172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:12.891731 1172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:12.891788 1172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:12.891842 1172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:12.891868 1172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:12.898334 1172 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:39447
I20260924 17:43:12.898411 1300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:39447 every 8 connection(s)
I20260924 17:43:12.898751 1172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:12 dist-test-slave-r7hf krb5kdc[1065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271792, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:12.907444 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1172
I20260924 17:43:12.907554 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:12.907752 1058 external_mini_cluster.cc:1428] Setting key 98c2e222ab608093e5290eb1366f0e3f
I20260924 17:43:12.908802 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:12.899547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:59337 (local address 127.1.8.190:33633)
0924 17:43:12.899618 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:12.899621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:12.899925 (+ 304us) server_negotiation.cc:407] Connection header received
0924 17:43:12.900427 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:12.900428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:12.900438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:12.900488 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:12.901436 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:12.901935 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:12.902983 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:12.903169 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:12.903222 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:12.905584 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:12.905586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:12.905586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:12.905593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:12.907132 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:12.907703 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:12.907707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:12.907707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:12.907775 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:12.908097 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:12.908098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:12.908098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:12.908229 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:12.908310 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:12.908614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 17:43:12.908704 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:12.909026 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:12.899535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:33633 (local address 127.1.8.129:59337)
0924 17:43:12.899699 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:12.899738 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:12.900328 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:12.900555 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:12.900558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:12.901141 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:12.901355 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:12.901360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:12.902036 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:12.902037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:12.902906 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:12.902909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:12.903243 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:12.903244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:12.903346 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:12.903902 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:12.903911 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:12.905389 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:12.907266 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:12.907274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:12.907277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:12.907622 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:12.907862 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:12.907864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:12.907864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:12.908019 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:12.908301 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:12.908307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:12.908477 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 17:43:12.908595 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 17:43:12.908772 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260924 17:43:12.909507 1301 heartbeater.cc:347] Connected to a master server at 127.1.8.190:33633
I20260924 17:43:12.909621 1301 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:12.909807 1301 heartbeater.cc:511] Master 127.1.8.190:33633 requested a full tablet report, sending...
I20260924 17:43:12.910460 1110 ts_manager.cc:194] Registered new tserver with Master: 0635fe139f994bb3808c0a1691e7f1a0 (127.1.8.129:39447)
I20260924 17:43:12.911473 1110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:59337
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:12.935420 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:33633
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:13.020710 1306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:13.020941 1306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:13.020967 1306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:13.022138 1306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:13.022178 1306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:13.022214 1306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:13.023694 1306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:33633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:13.023900 1306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:13.024091 1306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:13.029577 1312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:13.029608 1306 server_base.cc:1034] running on GCE node
W20260924 17:43:13.029577 1314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:13.029577 1311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:13.030069 1306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:13.030349 1306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:13.031608 1306 hybrid_clock.cc:648] HybridClock initialized: now 1790271793031595 us; error 36 us; skew 500 ppm
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:13.034608 1306 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:13.035140 1306 webserver.cc:460] Webserver started at http://127.1.8.130:41029/ using document root <none> and password file <none>
I20260924 17:43:13.035382 1306 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:13.035468 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:13.035624 1306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:13.036715 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "85a9ad104187466f8b07a14078d24f84"
format_stamp: "Formatted at 2026-09-24 17:43:13 on dist-test-slave-r7hf"
server_key: "bdfb2070265fa57952787939f98fe347"
server_key_iv: "5941473365f372cbb232dbace9f96d6f"
server_key_version: "encryptionkey@0"
I20260924 17:43:13.037106 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "85a9ad104187466f8b07a14078d24f84"
format_stamp: "Formatted at 2026-09-24 17:43:13 on dist-test-slave-r7hf"
server_key: "bdfb2070265fa57952787939f98fe347"
server_key_iv: "5941473365f372cbb232dbace9f96d6f"
server_key_version: "encryptionkey@0"
I20260924 17:43:13.038551 1306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:13.039340 1321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:13.039502 1306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:13.039609 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "85a9ad104187466f8b07a14078d24f84"
format_stamp: "Formatted at 2026-09-24 17:43:13 on dist-test-slave-r7hf"
server_key: "bdfb2070265fa57952787939f98fe347"
server_key_iv: "5941473365f372cbb232dbace9f96d6f"
server_key_version: "encryptionkey@0"
I20260924 17:43:13.039685 1306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:13.050134 1306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:13.052423 1306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:13.052601 1306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:13.052855 1306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:13.053188 1306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:13.053246 1306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:13.053292 1306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:13.053324 1306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:13.060371 1306 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:46739
I20260924 17:43:13.060410 1434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:46739 every 8 connection(s)
I20260924 17:43:13.060768 1306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:13.070183 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:13.061579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:43483 (local address 127.1.8.190:33633)
0924 17:43:13.061626 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:13.061628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:13.061872 (+ 244us) server_negotiation.cc:407] Connection header received
0924 17:43:13.062325 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:13.062327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:13.062338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:13.062424 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:13.063453 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.063950 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.064971 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.065157 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.065212 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:13.067386 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:13.067388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:13.067389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:13.067396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:13.068690 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.069138 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.069139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.069139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.069188 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.069459 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.069460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.069460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.069566 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:13.069617 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:13.070033 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 17:43:13.070103 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:13.070379 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:13.061547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:33633 (local address 127.1.8.130:43483)
0924 17:43:13.061702 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:13.061740 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:13.062227 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:13.062544 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:13.062547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:13.063158 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:13.063360 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:13.063365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.064030 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.064030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:13.064883 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:13.064885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.065232 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.065232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:13.065333 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:13.065840 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:13.065848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:13.067291 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:13.068769 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:13.068775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:13.068777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:13.069068 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:13.069269 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:13.069270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:13.069271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:13.069391 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:13.069668 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:13.069673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:13.069882 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 17:43:13.069998 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 17:43:13.070135 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260924 17:43:13.070905 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1306
I20260924 17:43:13.071027 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:13.071187 1435 heartbeater.cc:347] Connected to a master server at 127.1.8.190:33633
I20260924 17:43:13.071228 1058 external_mini_cluster.cc:1428] Setting key 97d10a5a0c758f5378525313d3a5c96d
I20260924 17:43:13.071300 1435 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:13.071506 1435 heartbeater.cc:511] Master 127.1.8.190:33633 requested a full tablet report, sending...
I20260924 17:43:13.071940 1109 ts_manager.cc:194] Registered new tserver with Master: 85a9ad104187466f8b07a14078d24f84 (127.1.8.130:46739)
I20260924 17:43:13.072335 1109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:43483
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:13.098361 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:33633
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:13.183962 1440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:13.184196 1440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:13.184224 1440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:13.185426 1440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:13.185470 1440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:13.185500 1440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:13.187064 1440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:33633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:13.187280 1440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:13.187475 1440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:13.192812 1448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:13.193019 1440 server_base.cc:1034] running on GCE node
W20260924 17:43:13.192818 1445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:13.192812 1446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:13.193423 1440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:13.193653 1440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:13.194810 1440 hybrid_clock.cc:648] HybridClock initialized: now 1790271793194789 us; error 39 us; skew 500 ppm
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:13.197520 1440 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:13.198043 1440 webserver.cc:460] Webserver started at http://127.1.8.131:34973/ using document root <none> and password file <none>
I20260924 17:43:13.198283 1440 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:13.198331 1440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:13.198535 1440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:13.199720 1440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "a37a1fe1e45d435bbcc076b6ff8e30b1"
format_stamp: "Formatted at 2026-09-24 17:43:13 on dist-test-slave-r7hf"
server_key: "25f33ad0899e358244ccb67babcc0150"
server_key_iv: "3dbe41fa33590e7b0edcd483be576a28"
server_key_version: "encryptionkey@0"
I20260924 17:43:13.200057 1440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "a37a1fe1e45d435bbcc076b6ff8e30b1"
format_stamp: "Formatted at 2026-09-24 17:43:13 on dist-test-slave-r7hf"
server_key: "25f33ad0899e358244ccb67babcc0150"
server_key_iv: "3dbe41fa33590e7b0edcd483be576a28"
server_key_version: "encryptionkey@0"
I20260924 17:43:13.201530 1440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:13.202329 1455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:13.202548 1440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:13.202687 1440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "a37a1fe1e45d435bbcc076b6ff8e30b1"
format_stamp: "Formatted at 2026-09-24 17:43:13 on dist-test-slave-r7hf"
server_key: "25f33ad0899e358244ccb67babcc0150"
server_key_iv: "3dbe41fa33590e7b0edcd483be576a28"
server_key_version: "encryptionkey@0"
I20260924 17:43:13.202769 1440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:13.215710 1440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:13.217818 1440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:13.218007 1440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:13.218259 1440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:13.218705 1440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:13.218772 1440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:13.218837 1440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:13.218878 1440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:13.226847 1440 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:41165
I20260924 17:43:13.226902 1568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:41165 every 8 connection(s)
I20260924 17:43:13.227319 1440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:13.233872 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1440
I20260924 17:43:13.233973 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:13.234128 1058 external_mini_cluster.cc:1428] Setting key 0fd910faa3b41fa86ee69c5181e62b7a
I20260924 17:43:13.237635 1150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:13.228154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:54247 (local address 127.1.8.190:33633)
0924 17:43:13.228242 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:13.228244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:13.228424 (+ 180us) server_negotiation.cc:407] Connection header received
0924 17:43:13.228896 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:13.228897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:13.228907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:13.228987 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:13.229978 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.230613 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.231652 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.231905 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.231992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:13.234483 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:13.234485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:13.234485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:13.234493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:13.236144 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.236607 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.236608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.236609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.236677 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.236943 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.236944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.236944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.237074 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:13.237151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:13.237444 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 17:43:13.237533 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:43:13.237866 1571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:13.228119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:33633 (local address 127.1.8.131:54247)
0924 17:43:13.228290 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:13.228328 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:13.228811 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:13.229044 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:13.229048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:13.229635 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:13.229842 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:13.229847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.230713 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.230714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:13.231571 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:13.231574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.231996 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.231996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:13.232098 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:13.232624 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:13.232633 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:13.234324 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:13.236255 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:13.236263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:13.236265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:13.236525 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:13.236747 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:13.236748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:13.236748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:13.236872 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:13.237142 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:13.237148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:13.237302 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 17:43:13.237425 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 17:43:13.237610 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260924 17:43:13.238534 1569 heartbeater.cc:347] Connected to a master server at 127.1.8.190:33633
I20260924 17:43:13.238633 1569 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:13.238894 1569 heartbeater.cc:511] Master 127.1.8.190:33633 requested a full tablet report, sending...
I20260924 17:43:13.239277 1109 ts_manager.cc:194] Registered new tserver with Master: a37a1fe1e45d435bbcc076b6ff8e30b1 (127.1.8.131:41165)
I20260924 17:43:13.239667 1109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:54247
I20260924 17:43:13.246704 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:13.272528 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:13.263980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.1.8.129:39447)
0924 17:43:13.264153 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:13.264160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:13.264177 (+ 17us) server_negotiation.cc:407] Connection header received
0924 17:43:13.264204 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:13.264206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:13.264220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:13.264288 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:13.264962 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.265699 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.266759 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.267101 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.267164 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:13.269462 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:13.269472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:13.269473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:13.269488 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:13.271247 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.271634 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.271644 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.271644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.271713 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.271923 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.271924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.271924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.272070 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:13.272229 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:13.272298 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 17:43:13.272401 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
W20260924 17:43:13.272984 1215 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54220
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:13.296976 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:13.289011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.1.8.129:39447)
0924 17:43:13.289080 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:13.289082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:13.289168 (+ 86us) server_negotiation.cc:407] Connection header received
0924 17:43:13.289310 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:13.289312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:13.289327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:13.289404 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:13.290034 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.290591 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.291351 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:13.291566 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:13.291656 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:13.293980 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:13.293982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:13.293982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:13.293994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:13.295778 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.296144 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.296145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.296145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.296207 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:13.296422 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:13.296423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:13.296423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:13.296535 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:13.296666 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:13.296763 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 17:43:13.296851 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:43:13.297609 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1172
I20260924 17:43:13.303324 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1306
I20260924 17:43:13.308547 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1440
I20260924 17:43:13.313443 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1078
2026-09-24T17:43:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1928 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1583](info): set up 2 sockets
Sep 24 17:43:13 dist-test-slave-r7hf krb5kdc[1583](info): commencing operation
krb5kdc: starting...
W20260924 17:43:14.366202 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:14 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:14.507054 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39579
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:14.589823 1596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:14.590067 1596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:14.590091 1596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:14.591272 1596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:14.591315 1596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:14.591329 1596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:14.591342 1596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:14.591361 1596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:14.592849 1596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:14.593076 1596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:14.593266 1596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:14.598686 1601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:14.598695 1602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:14.598687 1604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:14.599803 1596 server_base.cc:1034] running on GCE node
I20260924 17:43:14.600139 1596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:14.600433 1596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:14.601591 1596 hybrid_clock.cc:648] HybridClock initialized: now 1790271794601556 us; error 49 us; skew 500 ppm
Sep 24 17:43:14 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:14.604490 1596 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:14.605041 1596 webserver.cc:460] Webserver started at http://127.1.8.190:45179/ using document root <none> and password file <none>
I20260924 17:43:14.605283 1596 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:14.605357 1596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:14.605518 1596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:14.606734 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "b6dafd946c58417d8816a4c37492f4df"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "e4df3cdf6cb99585dc4828ad70f3eafa"
server_key_iv: "e6c6fd4bd717248947c3c20a36fe5d23"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.607125 1596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "b6dafd946c58417d8816a4c37492f4df"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "e4df3cdf6cb99585dc4828ad70f3eafa"
server_key_iv: "e6c6fd4bd717248947c3c20a36fe5d23"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.608575 1596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:14.609413 1611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:14.609599 1596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:14.609712 1596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "b6dafd946c58417d8816a4c37492f4df"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "e4df3cdf6cb99585dc4828ad70f3eafa"
server_key_iv: "e6c6fd4bd717248947c3c20a36fe5d23"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.609792 1596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:14.628615 1596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:14.631385 1596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:14.631578 1596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:14.635793 1596 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39579
I20260924 17:43:14.635874 1663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39579 every 8 connection(s)
I20260924 17:43:14.636272 1596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:14.637250 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:14.639803 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df: Bootstrap starting.
I20260924 17:43:14.640452 1664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:14.640810 1664 log.cc:824] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:14.641609 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df: No bootstrap required, opened a new log
I20260924 17:43:14.642705 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1596
I20260924 17:43:14.642825 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:14.642997 1058 external_mini_cluster.cc:1428] Setting key cef516f54693bfaff66202875ad9c0d0
I20260924 17:43:14.643953 1664 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dafd946c58417d8816a4c37492f4df" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39579 } }
I20260924 17:43:14.644140 1664 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:14.644192 1664 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6dafd946c58417d8816a4c37492f4df, State: Initialized, Role: FOLLOWER
I20260924 17:43:14.644356 1664 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dafd946c58417d8816a4c37492f4df" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39579 } }
I20260924 17:43:14.644467 1664 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:14.644491 1664 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:14.644556 1664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:14.645797 1664 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dafd946c58417d8816a4c37492f4df" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39579 } }
I20260924 17:43:14.645915 1664 leader_election.cc:302] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6dafd946c58417d8816a4c37492f4df; no voters:
I20260924 17:43:14.646075 1664 leader_election.cc:288] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:14.646153 1669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:14.646327 1669 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [term 1 LEADER]: Becoming Leader. State: Replica: b6dafd946c58417d8816a4c37492f4df, State: Running, Role: LEADER
I20260924 17:43:14.646373 1664 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:14.646464 1669 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dafd946c58417d8816a4c37492f4df" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39579 } }
I20260924 17:43:14.648061 1670 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6dafd946c58417d8816a4c37492f4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dafd946c58417d8816a4c37492f4df" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39579 } } }
I20260924 17:43:14.648350 1670 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [sys.catalog]: This master's current role is: LEADER
W20260924 17:43:14.648578 1683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:14.648648 1683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:14.648288 1671 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6dafd946c58417d8816a4c37492f4df. Latest consensus state: current_term: 1 leader_uuid: "b6dafd946c58417d8816a4c37492f4df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6dafd946c58417d8816a4c37492f4df" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39579 } } }
I20260924 17:43:14.648752 1671 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:14.648906 1685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:14.649081 1685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:43:14 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:14.650905 1685 catalog_manager.cc:1347] Generated new cluster ID: 91b1021b0af14122b43bd603f782df30
I20260924 17:43:14.650954 1685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:14.655942 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:14.643595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.1.8.190:39579)
0924 17:43:14.643828 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:14.643839 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:14.643885 (+ 46us) server_negotiation.cc:407] Connection header received
0924 17:43:14.644010 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:14.644018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:14.644077 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:14.644260 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:14.645009 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:14.646488 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:14.647393 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:14.648128 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:14.648237 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:14.651346 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:14.651351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:14.651351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:14.651367 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:14.653655 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:14.654104 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:14.654108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:14.654108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:14.654259 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:14.654499 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:14.654499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:14.654500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:14.654875 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:14.655082 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:14.655163 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 17:43:14.655476 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260924 17:43:14.678856 1685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:14.679512 1685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:14.682366 1685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6dafd946c58417d8816a4c37492f4df: Generated new TSK 0
I20260924 17:43:14.682617 1685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:14.718174 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39579
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:14.805541 1690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:14.805796 1690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:14.805826 1690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:14.807178 1690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:14.807224 1690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:14.807255 1690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:14.808964 1690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:14.809196 1690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:14.809408 1690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:14.815249 1695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:14.815250 1698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:14.815250 1696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:14.815670 1690 server_base.cc:1034] running on GCE node
I20260924 17:43:14.815944 1690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:14.816218 1690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:14.817418 1690 hybrid_clock.cc:648] HybridClock initialized: now 1790271794817401 us; error 38 us; skew 500 ppm
Sep 24 17:43:14 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:14.820277 1690 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:14.820782 1690 webserver.cc:460] Webserver started at http://127.1.8.129:36311/ using document root <none> and password file <none>
I20260924 17:43:14.821055 1690 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:14.821105 1690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:14.821249 1690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:14.822512 1690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "7f245ab4f95d44f993c3c5bb62202b2e"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "3fa798296c6999475cdacbee51a96e30"
server_key_iv: "64e509ac099f3fb2d295741e101d590b"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.822908 1690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "7f245ab4f95d44f993c3c5bb62202b2e"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "3fa798296c6999475cdacbee51a96e30"
server_key_iv: "64e509ac099f3fb2d295741e101d590b"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.824440 1690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:14.825466 1705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:14.825685 1690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:14.825836 1690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "7f245ab4f95d44f993c3c5bb62202b2e"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "3fa798296c6999475cdacbee51a96e30"
server_key_iv: "64e509ac099f3fb2d295741e101d590b"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.825904 1690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:14.842365 1690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:14.844689 1690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:14.844906 1690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:14.845109 1690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:14.845474 1690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:14.845522 1690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:14.845546 1690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:14.845557 1690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:14.852007 1690 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:37589
I20260924 17:43:14.852082 1818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:37589 every 8 connection(s)
I20260924 17:43:14.852440 1690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:43:14.853600 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1690
I20260924 17:43:14.853806 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:14.854018 1058 external_mini_cluster.cc:1428] Setting key 158db2034643b36d76f0e1c47b83441a
Sep 24 17:43:14 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:43:14.864602 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:14.853352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:44889 (local address 127.1.8.190:39579)
0924 17:43:14.853440 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:14.853442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:14.853682 (+ 240us) server_negotiation.cc:407] Connection header received
0924 17:43:14.855042 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:14.855042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:14.855056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:14.855097 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:14.856107 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:14.857041 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:14.858201 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:14.858593 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:14.858677 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:14.861164 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:14.861166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:14.861166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:14.861188 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:14.862958 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:14.863439 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:14.863440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:14.863441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:14.863502 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:14.863855 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:14.863856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:14.863857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:14.863986 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:14.864069 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:14.864391 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 17:43:14.864504 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 17:43:14.864849 1821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:14.853299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39579 (local address 127.1.8.129:44889)
0924 17:43:14.853504 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:14.853559 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:14.854906 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:14.855219 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:14.855224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:14.855657 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:14.855972 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:14.855978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:14.857191 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:14.857193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:14.858097 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:14.858099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:14.858732 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:14.858733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:14.858864 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:14.859537 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:14.859547 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:14.861048 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:14.863045 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:14.863053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:14.863056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:14.863329 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:14.863626 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:14.863627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:14.863627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:14.863763 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:14.864073 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:14.864081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:14.864247 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 17:43:14.864368 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 17:43:14.864565 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
I20260924 17:43:14.865469 1819 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39579
I20260924 17:43:14.865594 1819 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:14.865885 1819 heartbeater.cc:511] Master 127.1.8.190:39579 requested a full tablet report, sending...
I20260924 17:43:14.866680 1628 ts_manager.cc:194] Registered new tserver with Master: 7f245ab4f95d44f993c3c5bb62202b2e (127.1.8.129:37589)
I20260924 17:43:14.867389 1628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:44889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:14.882751 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39579
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:14.967307 1824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:14.967593 1824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:14.967638 1824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:14.968910 1824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:14.968992 1824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:14.969043 1824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:14.970677 1824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:14.970999 1824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:14.971246 1824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:14.976647 1829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:14.976809 1830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:14.976807 1824 server_base.cc:1034] running on GCE node
W20260924 17:43:14.976647 1832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:14.977195 1824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:14.977480 1824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:14.978652 1824 hybrid_clock.cc:648] HybridClock initialized: now 1790271794978626 us; error 45 us; skew 500 ppm
Sep 24 17:43:14 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:14.981320 1824 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:14.981887 1824 webserver.cc:460] Webserver started at http://127.1.8.130:43893/ using document root <none> and password file <none>
I20260924 17:43:14.982141 1824 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:14.982211 1824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:14.982365 1824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:14.983505 1824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "72b023766533445d8d67ee424a2e7f26"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "634c30f538ed2c7d1be7ca823f3495b7"
server_key_iv: "aa4ff0a96ccc52bdad86385195491812"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.983894 1824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "72b023766533445d8d67ee424a2e7f26"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "634c30f538ed2c7d1be7ca823f3495b7"
server_key_iv: "aa4ff0a96ccc52bdad86385195491812"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.985347 1824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:14.986188 1839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:14.986456 1824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:14.986558 1824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "72b023766533445d8d67ee424a2e7f26"
format_stamp: "Formatted at 2026-09-24 17:43:14 on dist-test-slave-r7hf"
server_key: "634c30f538ed2c7d1be7ca823f3495b7"
server_key_iv: "aa4ff0a96ccc52bdad86385195491812"
server_key_version: "encryptionkey@0"
I20260924 17:43:14.986645 1824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:15.007181 1824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:15.009390 1824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:15.009573 1824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:15.009822 1824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:15.010154 1824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:15.010213 1824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:15.010258 1824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:15.010283 1824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:15.016613 1824 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39763
I20260924 17:43:15.016659 1952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39763 every 8 connection(s)
I20260924 17:43:15.017011 1824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:15.018597 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1824
I20260924 17:43:15.018693 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:15.018857 1058 external_mini_cluster.cc:1428] Setting key 49661adf12c7065731cde0a8151ebf9d
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:43:15.029206 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.017892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:45849 (local address 127.1.8.190:39579)
0924 17:43:15.018019 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.018022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.018226 (+ 204us) server_negotiation.cc:407] Connection header received
0924 17:43:15.019659 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.019660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.019672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.019720 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:15.020661 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.021279 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.022994 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.023192 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.023262 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.026073 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:15.026074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:15.026074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:15.026084 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:15.027651 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.028104 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.028106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.028106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.028168 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.028451 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.028452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.028452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.028578 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:15.028662 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.029008 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.029105 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 17:43:15.029436 1955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.017877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39579 (local address 127.1.8.130:45849)
0924 17:43:15.018076 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.018128 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.019539 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.019821 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.019826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.020257 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:15.020556 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.020562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.021362 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.021363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.022879 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.022882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.023300 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.023301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.023483 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.024179 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:15.024193 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:15.025958 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:15.027768 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:15.027777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:15.027780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:15.028023 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:15.028236 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:15.028237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:15.028237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:15.028373 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:15.028656 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:15.028663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:15.028850 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.028988 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.029168 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":105,"threads_started":1}
I20260924 17:43:15.030082 1953 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39579
I20260924 17:43:15.030177 1953 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:15.030453 1953 heartbeater.cc:511] Master 127.1.8.190:39579 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
I20260924 17:43:15.030923 1628 ts_manager.cc:194] Registered new tserver with Master: 72b023766533445d8d67ee424a2e7f26 (127.1.8.130:39763)
I20260924 17:43:15.031347 1628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:45849
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:15.050992 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39579
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:15.137307 1958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:15.137542 1958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:15.137568 1958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:15.138854 1958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:15.138908 1958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:15.138939 1958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:15.140604 1958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:15.140923 1958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:15.141196 1958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:15.146639 1964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:15.146669 1963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:15.146639 1966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:15.147032 1958 server_base.cc:1034] running on GCE node
I20260924 17:43:15.147267 1958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:15.147585 1958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:15.148931 1958 hybrid_clock.cc:648] HybridClock initialized: now 1790271795148911 us; error 36 us; skew 500 ppm
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271795, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:15.151683 1958 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:15.152251 1958 webserver.cc:460] Webserver started at http://127.1.8.131:46585/ using document root <none> and password file <none>
I20260924 17:43:15.152506 1958 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:15.152583 1958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:15.152736 1958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:15.153858 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "e12225313eae42f28ca1558101794dbe"
format_stamp: "Formatted at 2026-09-24 17:43:15 on dist-test-slave-r7hf"
server_key: "b743a1eb2e86d9131db635422d61b643"
server_key_iv: "bdebc405f0c47a0fd780ddb59b4db32b"
server_key_version: "encryptionkey@0"
I20260924 17:43:15.154229 1958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "e12225313eae42f28ca1558101794dbe"
format_stamp: "Formatted at 2026-09-24 17:43:15 on dist-test-slave-r7hf"
server_key: "b743a1eb2e86d9131db635422d61b643"
server_key_iv: "bdebc405f0c47a0fd780ddb59b4db32b"
server_key_version: "encryptionkey@0"
I20260924 17:43:15.155802 1958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:15.156797 1973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:15.157059 1958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:15.157177 1958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "e12225313eae42f28ca1558101794dbe"
format_stamp: "Formatted at 2026-09-24 17:43:15 on dist-test-slave-r7hf"
server_key: "b743a1eb2e86d9131db635422d61b643"
server_key_iv: "bdebc405f0c47a0fd780ddb59b4db32b"
server_key_version: "encryptionkey@0"
I20260924 17:43:15.157289 1958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:15.168308 1958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:15.170859 1958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:15.171037 1958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:15.171294 1958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:15.171631 1958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:15.171687 1958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:15.171736 1958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:15.171759 1958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:15.177785 1958 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:39231
I20260924 17:43:15.177881 2086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:39231 every 8 connection(s)
I20260924 17:43:15.178221 1958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271795, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:15.187044 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 1958
I20260924 17:43:15.187170 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:15.187338 1058 external_mini_cluster.cc:1428] Setting key 9d698bc104acf339379c1f68074b9c69
I20260924 17:43:15.188659 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.179130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:44975 (local address 127.1.8.190:39579)
0924 17:43:15.179216 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.179219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.179417 (+ 198us) server_negotiation.cc:407] Connection header received
0924 17:43:15.180471 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.180473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.180487 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.180528 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:15.181198 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.181712 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.183065 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.183277 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.183370 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.185702 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:15.185704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:15.185704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:15.185713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:15.187103 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.187626 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.187627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.187628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.187689 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.187975 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.187976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.187976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.188101 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:15.188178 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.188477 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.188567 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:43:15.188897 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.179099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39579 (local address 127.1.8.131:44975)
0924 17:43:15.179284 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.179328 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.180337 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.180624 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.180628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.180899 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:15.181096 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.181100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.181848 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.181849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.182970 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.182977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.183379 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.183380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.183514 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.184052 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:15.184062 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:15.185593 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:15.187192 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:15.187200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:15.187203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:15.187536 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:15.187758 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:15.187759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:15.187759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:15.187897 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:15.188168 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:15.188176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:15.188339 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.188456 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.188627 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260924 17:43:15.189424 2087 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39579
I20260924 17:43:15.189534 2087 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:15.189740 2087 heartbeater.cc:511] Master 127.1.8.190:39579 requested a full tablet report, sending...
I20260924 17:43:15.190131 1628 ts_manager.cc:194] Registered new tserver with Master: e12225313eae42f28ca1558101794dbe (127.1.8.131:39231)
I20260924 17:43:15.190580 1628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:44975
I20260924 17:43:15.199407 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:43:15.207636 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.201078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.1.8.190:39579)
0924 17:43:15.201167 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.201169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.201220 (+ 51us) server_negotiation.cc:407] Connection header received
0924 17:43:15.201308 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.201309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.201320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.201372 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:15.201944 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.202517 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.203246 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.203455 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.203541 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.204829 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:15.204830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:15.204830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:15.204839 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:15.206299 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.206733 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.206735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.206735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.206805 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.207009 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.207010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.207010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.207182 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:15.207291 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.207441 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.207526 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 17:43:15.210461 1628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:15.212237 1628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:15.219413 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.214850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:37589 (local address 127.0.0.1:55604)
0924 17:43:15.215025 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.215069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.215141 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.215782 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.215783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.215785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:15.215829 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.215831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.216819 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.216820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.218164 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.218167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.219039 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.219040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.219130 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.219144 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.219215 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.219277 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260924 17:43:15.219538 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.215062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.1.8.129:37589)
0924 17:43:15.215555 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.215559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.215567 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:15.215599 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.215601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.215639 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.215767 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:15.215944 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.216697 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.218249 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.218930 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.219036 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.219103 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.219403 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.219453 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":105,"threads_started":1}
I20260924 17:43:15.219681 2100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.215009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:39231 (local address 127.0.0.1:56604)
0924 17:43:15.215254 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.215265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.215320 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.215680 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.215681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.215684 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:15.215773 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.215776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.217180 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.217181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.218256 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.218257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.219182 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.219183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.219242 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.219245 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.219302 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.219335 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260924 17:43:15.219444 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.215287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.1.8.131:39231)
0924 17:43:15.215514 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.215518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.215527 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:15.215561 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.215563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.215579 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.215855 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:15.215932 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.217056 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.218319 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.219071 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.219178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.219244 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.219318 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.219356 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260924 17:43:15.220273 1752 tablet_service.cc:1467] Processing CreateTablet for tablet 928297b2d51d419fb15d6f3ef500c393 (DEFAULT_TABLE table=test-table [id=2767ffe4520a48b384c5b25f018e810c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:15.220693 1752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928297b2d51d419fb15d6f3ef500c393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:15.220722 2099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.215068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39763 (local address 127.0.0.1:48344)
0924 17:43:15.215398 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.215407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.215442 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.215666 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.215669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.215671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:15.215760 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.215763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.218338 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.218339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.219413 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.219416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.220450 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.220451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.220523 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.220528 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.220575 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.220621 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260924 17:43:15.220664 2021 tablet_service.cc:1467] Processing CreateTablet for tablet 928297b2d51d419fb15d6f3ef500c393 (DEFAULT_TABLE table=test-table [id=2767ffe4520a48b384c5b25f018e810c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:15.221009 2021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928297b2d51d419fb15d6f3ef500c393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:15.221024 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.215283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.1.8.130:39763)
0924 17:43:15.215465 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.215470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.215481 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:15.215527 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.215529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.215547 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.215638 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:15.215917 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.217065 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.219511 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.220295 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.220364 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.220415 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.220898 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.220946 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260924 17:43:15.221711 1887 tablet_service.cc:1467] Processing CreateTablet for tablet 928297b2d51d419fb15d6f3ef500c393 (DEFAULT_TABLE table=test-table [id=2767ffe4520a48b384c5b25f018e810c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:15.222033 1887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 928297b2d51d419fb15d6f3ef500c393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:15.223726 2104 tablet_bootstrap.cc:492] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: Bootstrap starting.
I20260924 17:43:15.224378 2105 tablet_bootstrap.cc:492] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: Bootstrap starting.
I20260924 17:43:15.224541 2104 tablet_bootstrap.cc:654] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:15.225185 2104 log.cc:824] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:15.225430 2105 tablet_bootstrap.cc:654] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:15.225991 2105 log.cc:824] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:15.226151 2104 tablet_bootstrap.cc:492] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: No bootstrap required, opened a new log
I20260924 17:43:15.226229 2104 ts_tablet_manager.cc:1397] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:15.226639 2108 tablet_bootstrap.cc:492] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: Bootstrap starting.
I20260924 17:43:15.226934 2105 tablet_bootstrap.cc:492] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: No bootstrap required, opened a new log
I20260924 17:43:15.227013 2105 ts_tablet_manager.cc:1397] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:15.227274 2108 tablet_bootstrap.cc:654] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:15.227766 2108 log.cc:824] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:15.228425 2104 raft_consensus.cc:348] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.228621 2104 raft_consensus.cc:374] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:15.228662 2104 raft_consensus.cc:729] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e12225313eae42f28ca1558101794dbe, State: Initialized, Role: FOLLOWER
I20260924 17:43:15.228817 2104 consensus_queue.cc:260] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.229123 2104 ts_tablet_manager.cc:1428] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:15.229256 2087 heartbeater.cc:503] Master 127.1.8.190:39579 was elected leader, sending a full tablet report...
I20260924 17:43:15.229158 2105 raft_consensus.cc:348] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.229337 2105 raft_consensus.cc:374] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:15.229377 2105 raft_consensus.cc:729] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f245ab4f95d44f993c3c5bb62202b2e, State: Initialized, Role: FOLLOWER
I20260924 17:43:15.229714 2105 consensus_queue.cc:260] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.230023 2105 ts_tablet_manager.cc:1428] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:15.230047 2108 tablet_bootstrap.cc:492] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: No bootstrap required, opened a new log
I20260924 17:43:15.230139 2108 ts_tablet_manager.cc:1397] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 17:43:15.230175 1819 heartbeater.cc:503] Master 127.1.8.190:39579 was elected leader, sending a full tablet report...
I20260924 17:43:15.232788 2108 raft_consensus.cc:348] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.232980 2108 raft_consensus.cc:374] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:15.233035 2108 raft_consensus.cc:729] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72b023766533445d8d67ee424a2e7f26, State: Initialized, Role: FOLLOWER
I20260924 17:43:15.233170 2108 consensus_queue.cc:260] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.233690 2108 ts_tablet_manager.cc:1428] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 17:43:15.233700 1953 heartbeater.cc:503] Master 127.1.8.190:39579 was elected leader, sending a full tablet report...
W20260924 17:43:15.267647 1954 tablet.cc:2369] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:43:15.353245 1820 tablet.cc:2369] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:43:15.382283 2111 raft_consensus.cc:482] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:15.382537 2111 raft_consensus.cc:504] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.383071 2111 leader_election.cc:288] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e12225313eae42f28ca1558101794dbe (127.1.8.131:39231), 72b023766533445d8d67ee424a2e7f26 (127.1.8.130:39763)
I20260924 17:43:15.386467 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.383427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:48027 (local address 127.1.8.130:39763)
0924 17:43:15.383479 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.383488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.383553 (+ 65us) server_negotiation.cc:407] Connection header received
0924 17:43:15.383647 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.383648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.383679 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.383740 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:15.384013 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.384486 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.385681 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.386046 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.386102 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.386111 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.386315 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.386365 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:43:15.386502 2114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.383312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39763 (local address 127.1.8.129:48027)
0924 17:43:15.383446 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.383460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.383560 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.383850 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.383851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.383854 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:15.383935 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.383937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.384585 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.384586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.385582 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.385585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.386137 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.386139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.386227 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.386232 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.386288 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.386369 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":51,"thread_start_us":75,"threads_started":1}
I20260924 17:43:15.386620 2113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.383312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:39231 (local address 127.1.8.129:59653)
0924 17:43:15.383450 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:15.383462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:15.383529 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:15.383701 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:15.383702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:15.383705 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:15.383790 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.383792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.384585 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.384586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.385495 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:15.385498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.386190 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.386191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:15.386264 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.386293 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 17:43:15.386377 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:43:15.386443 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":42,"thread_start_us":57,"threads_started":1}
I20260924 17:43:15.387058 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.383500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:59653 (local address 127.1.8.131:39231)
0924 17:43:15.383554 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.383556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.383569 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:43:15.383607 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.383608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.383622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.383686 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:15.383876 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.384487 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.385602 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.386067 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.386173 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.386183 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.386941 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.386983 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:43:15.387002 1907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928297b2d51d419fb15d6f3ef500c393" candidate_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b023766533445d8d67ee424a2e7f26" is_pre_election: true
I20260924 17:43:15.387230 1907 raft_consensus.cc:2454] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f245ab4f95d44f993c3c5bb62202b2e in term 0.
I20260924 17:43:15.387352 2041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928297b2d51d419fb15d6f3ef500c393" candidate_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12225313eae42f28ca1558101794dbe" is_pre_election: true
I20260924 17:43:15.387524 2041 raft_consensus.cc:2454] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f245ab4f95d44f993c3c5bb62202b2e in term 0.
I20260924 17:43:15.387514 1709 leader_election.cc:302] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72b023766533445d8d67ee424a2e7f26, 7f245ab4f95d44f993c3c5bb62202b2e; no voters:
I20260924 17:43:15.387667 2111 raft_consensus.cc:2781] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:15.387751 2111 raft_consensus.cc:482] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:15.387785 2111 raft_consensus.cc:3037] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:15.388569 2111 raft_consensus.cc:504] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.388744 2111 leader_election.cc:288] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [CANDIDATE]: Term 1 election: Requested vote from peers e12225313eae42f28ca1558101794dbe (127.1.8.131:39231), 72b023766533445d8d67ee424a2e7f26 (127.1.8.130:39763)
I20260924 17:43:15.388937 2041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928297b2d51d419fb15d6f3ef500c393" candidate_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e12225313eae42f28ca1558101794dbe"
I20260924 17:43:15.389022 2041 raft_consensus.cc:3037] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:15.389768 2041 raft_consensus.cc:2454] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f245ab4f95d44f993c3c5bb62202b2e in term 1.
I20260924 17:43:15.389930 1907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "928297b2d51d419fb15d6f3ef500c393" candidate_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72b023766533445d8d67ee424a2e7f26"
I20260924 17:43:15.389989 1708 leader_election.cc:302] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f245ab4f95d44f993c3c5bb62202b2e, e12225313eae42f28ca1558101794dbe; no voters:
I20260924 17:43:15.390022 1907 raft_consensus.cc:3037] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:15.390123 2111 raft_consensus.cc:2781] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:15.390317 2111 raft_consensus.cc:686] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [term 1 LEADER]: Becoming Leader. State: Replica: 7f245ab4f95d44f993c3c5bb62202b2e, State: Running, Role: LEADER
I20260924 17:43:15.390446 2111 consensus_queue.cc:237] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } }
I20260924 17:43:15.390970 1907 raft_consensus.cc:2454] T 928297b2d51d419fb15d6f3ef500c393 P 72b023766533445d8d67ee424a2e7f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f245ab4f95d44f993c3c5bb62202b2e in term 1.
I20260924 17:43:15.391429 1627 catalog_manager.cc:5495] T 928297b2d51d419fb15d6f3ef500c393 P 7f245ab4f95d44f993c3c5bb62202b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f245ab4f95d44f993c3c5bb62202b2e (127.1.8.129). New cstate: current_term: 1 leader_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f245ab4f95d44f993c3c5bb62202b2e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 37589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e12225313eae42f28ca1558101794dbe" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 39231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72b023766533445d8d67ee424a2e7f26" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39763 } health_report { overall_health: UNKNOWN } } }
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:15.410511 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.401993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.1.8.129:37589)
0924 17:43:15.402063 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.402065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.402262 (+ 197us) server_negotiation.cc:407] Connection header received
0924 17:43:15.402310 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.402311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.402325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.402386 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:15.403023 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.403564 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.404354 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.404576 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.404638 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.407304 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:15.407316 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:15.407316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:15.407324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:15.409191 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.409611 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.409615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.409615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.409693 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.409958 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.409958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.409959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.410110 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:15.410215 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.410304 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.410373 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 17:43:15.428992 2088 tablet.cc:2369] T 928297b2d51d419fb15d6f3ef500c393 P e12225313eae42f28ca1558101794dbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271795, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:15.437252 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.429042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.1.8.129:37589)
0924 17:43:15.429092 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.429095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.429180 (+ 85us) server_negotiation.cc:407] Connection header received
0924 17:43:15.429336 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.429337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.429347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.429401 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:15.430207 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.430895 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.431647 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.431891 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.431951 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.434555 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:15.434557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:15.434557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:15.434564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:15.436140 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.436525 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.436526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.436526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.436580 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.436757 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.436758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.436758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.436879 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:15.436975 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.437111 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.437176 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[1583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:15.462321 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:15.454199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.1.8.129:37589)
0924 17:43:15.454281 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:15.454284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:15.454475 (+ 191us) server_negotiation.cc:407] Connection header received
0924 17:43:15.454516 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:15.454517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:15.454528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:15.454587 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:15.455257 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.455720 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.456566 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:15.456745 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:15.456806 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:15.459465 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:15.459467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:15.459467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:15.459474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:15.461102 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.461502 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.461503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.461503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.461569 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:15.461833 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:15.461834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:15.461834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:15.461943 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:15.462045 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:15.462132 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 17:43:15.462199 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 17:43:15.462942 1730 tablet_service.cc:2908] Rejecting scan request for tablet 928297b2d51d419fb15d6f3ef500c393: Uninitialized: clean time has not yet been initialized
I20260924 17:43:15.464066 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1690
I20260924 17:43:15.469439 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1824
I20260924 17:43:15.474898 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1958
I20260924 17:43:15.479717 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 1596
2026-09-24T17:43:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2166 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[2129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[2129](info): set up 2 sockets
Sep 24 17:43:15 dist-test-slave-r7hf krb5kdc[2129](info): commencing operation
krb5kdc: starting...
W20260924 17:43:16.538118 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:16 dist-test-slave-r7hf krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:16.670177 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39371
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:16.754235 2142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:16.754547 2142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:16.754592 2142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:16.755841 2142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:16.755924 2142 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 17:43:16.755946 2142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:16.755966 2142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:16.755996 2142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:16.756021 2142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:16.757623 2142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39371
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:16.757942 2142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:16.758203 2142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:16.763424 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:16.763412 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:16.763542 2142 server_base.cc:1034] running on GCE node
W20260924 17:43:16.763384 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:16.763968 2142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:16.764314 2142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:16.765472 2142 hybrid_clock.cc:648] HybridClock initialized: now 1790271796765450 us; error 35 us; skew 500 ppm
Sep 24 17:43:16 dist-test-slave-r7hf krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271796, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:16.776116 2142 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:16.776656 2142 webserver.cc:460] Webserver started at http://127.1.8.190:34057/ using document root <none> and password file <none>
I20260924 17:43:16.777007 2142 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:16.777092 2142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:16.777257 2142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:16.778489 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "19c8d985696a42529c8d70e121eb4446"
format_stamp: "Formatted at 2026-09-24 17:43:16 on dist-test-slave-r7hf"
server_key: "8b1777b238d4ba1b4a94cf9dc5ab44e4"
server_key_iv: "71c58482ea02130bb56f2e13cddf708c"
server_key_version: "encryptionkey@0"
I20260924 17:43:16.778872 2142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "19c8d985696a42529c8d70e121eb4446"
format_stamp: "Formatted at 2026-09-24 17:43:16 on dist-test-slave-r7hf"
server_key: "8b1777b238d4ba1b4a94cf9dc5ab44e4"
server_key_iv: "71c58482ea02130bb56f2e13cddf708c"
server_key_version: "encryptionkey@0"
I20260924 17:43:16.780308 2142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:43:16.781167 2157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:16.781345 2142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:16.781463 2142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "19c8d985696a42529c8d70e121eb4446"
format_stamp: "Formatted at 2026-09-24 17:43:16 on dist-test-slave-r7hf"
server_key: "8b1777b238d4ba1b4a94cf9dc5ab44e4"
server_key_iv: "71c58482ea02130bb56f2e13cddf708c"
server_key_version: "encryptionkey@0"
I20260924 17:43:16.781544 2142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:16.800315 2142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:16.802928 2142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:16.803114 2142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:16.808041 2142 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39371
I20260924 17:43:16.808130 2219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39371 every 8 connection(s)
I20260924 17:43:16.808523 2142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:16.809245 2220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:16.811800 2220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: Bootstrap starting.
I20260924 17:43:16.812469 2220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:16.812811 2220 log.cc:824] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:16.813568 2220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: No bootstrap required, opened a new log
I20260924 17:43:16.815841 2220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c8d985696a42529c8d70e121eb4446" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39371 } }
I20260924 17:43:16.816030 2220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:16.816039 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 2142
I20260924 17:43:16.816069 2220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c8d985696a42529c8d70e121eb4446, State: Initialized, Role: FOLLOWER
I20260924 17:43:16.816164 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:16.816321 2220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c8d985696a42529c8d70e121eb4446" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39371 } }
I20260924 17:43:16.816381 1058 external_mini_cluster.cc:1428] Setting key a13d5d9812fe903160bee5b7ef816ece
I20260924 17:43:16.816417 2220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:16.816466 2220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:16.816522 2220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:16.817273 2220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c8d985696a42529c8d70e121eb4446" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39371 } }
I20260924 17:43:16.817399 2220 leader_election.cc:302] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c8d985696a42529c8d70e121eb4446; no voters:
I20260924 17:43:16.817564 2220 leader_election.cc:288] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:16.817589 2225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:16.817729 2225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [term 1 LEADER]: Becoming Leader. State: Replica: 19c8d985696a42529c8d70e121eb4446, State: Running, Role: LEADER
I20260924 17:43:16.817843 2220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:16.817988 2225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c8d985696a42529c8d70e121eb4446" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39371 } }
I20260924 17:43:16.818537 2227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c8d985696a42529c8d70e121eb4446. Latest consensus state: current_term: 1 leader_uuid: "19c8d985696a42529c8d70e121eb4446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c8d985696a42529c8d70e121eb4446" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39371 } } }
I20260924 17:43:16.818701 2227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:16.819154 2238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:16.819412 2226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c8d985696a42529c8d70e121eb4446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c8d985696a42529c8d70e121eb4446" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39371 } } }
I20260924 17:43:16.819489 2226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:16.819948 2238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:16.821683 2238 catalog_manager.cc:1347] Generated new cluster ID: adf17e5c403440fb9f97f9b5f51e4946
I20260924 17:43:16.821728 2238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:43:16 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:16.828114 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:16.816898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.1.8.190:39371)
0924 17:43:16.817095 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:16.817106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:16.817148 (+ 42us) server_negotiation.cc:407] Connection header received
0924 17:43:16.817249 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:16.817255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:16.817299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:16.817437 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:16.818204 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:16.819771 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:16.820653 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:16.821195 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:16.821336 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:16.824205 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:16.824210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:16.824210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:16.824225 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:16.826292 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:16.826776 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:16.826779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:16.826780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:16.826919 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:16.827146 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:16.827147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:16.827147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:16.827467 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:16.827615 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:16.827750 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 17:43:16.827896 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260924 17:43:16.830163 2238 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:16.830945 2238 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:16.835021 2238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: Generated new TSK 0
I20260924 17:43:16.835263 2238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:16.866566 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39371
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:16.953550 2246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:16.953830 2246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:16.953872 2246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:16.955149 2246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:16.955233 2246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:16.955272 2246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:43:16.955305 2246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:16.956952 2246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:16.957266 2246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:16.957513 2246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:16.962970 2252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:16.962988 2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:16.962981 2254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:16.963361 2246 server_base.cc:1034] running on GCE node
I20260924 17:43:16.963521 2246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:16.963735 2246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:16.964908 2246 hybrid_clock.cc:648] HybridClock initialized: now 1790271796964886 us; error 41 us; skew 500 ppm
Sep 24 17:43:16 dist-test-slave-r7hf krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271796, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:16.967842 2246 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:16.968392 2246 webserver.cc:460] Webserver started at http://127.1.8.129:46091/ using document root <none> and password file <none>
I20260924 17:43:16.968662 2246 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:16.968714 2246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:16.968816 2246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:16.970005 2246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "44727c74c5944ebe8429b0e77c79304e"
format_stamp: "Formatted at 2026-09-24 17:43:16 on dist-test-slave-r7hf"
server_key: "f0ca65c097a1144d1772989d84ea5e07"
server_key_iv: "b53bb87c8fc0f131bd5b4cd281038e40"
server_key_version: "encryptionkey@0"
I20260924 17:43:16.970382 2246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "44727c74c5944ebe8429b0e77c79304e"
format_stamp: "Formatted at 2026-09-24 17:43:16 on dist-test-slave-r7hf"
server_key: "f0ca65c097a1144d1772989d84ea5e07"
server_key_iv: "b53bb87c8fc0f131bd5b4cd281038e40"
server_key_version: "encryptionkey@0"
I20260924 17:43:16.971843 2246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:16.972916 2261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:16.973169 2246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:43:16.973287 2246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "44727c74c5944ebe8429b0e77c79304e"
format_stamp: "Formatted at 2026-09-24 17:43:16 on dist-test-slave-r7hf"
server_key: "f0ca65c097a1144d1772989d84ea5e07"
server_key_iv: "b53bb87c8fc0f131bd5b4cd281038e40"
server_key_version: "encryptionkey@0"
I20260924 17:43:16.973387 2246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:17.002591 2246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:17.005057 2246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:17.005188 2246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:17.005774 2246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:17.005851 2246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.005888 2246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:17.005918 2246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.013124 2246 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42449
I20260924 17:43:17.013213 2376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42449 every 8 connection(s)
I20260924 17:43:17.013595 2246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271796, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:17.017867 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.006530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49543 (local address 127.1.8.190:39371)
0924 17:43:17.006581 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.006584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.006878 (+ 294us) server_negotiation.cc:407] Connection header received
0924 17:43:17.007601 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.007603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.007619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.007698 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.008851 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.009726 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.011491 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.011900 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.011986 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.014937 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.014939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.014940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.014951 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.016506 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.016939 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.016940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.016940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.016992 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.017215 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.017216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.017216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.017359 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.017410 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.017669 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.017780 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:43:17.017982 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.006559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.1.8.129:49543)
0924 17:43:17.006753 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.006796 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.007492 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.007814 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.007820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.008364 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:17.008707 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.008715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.009855 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.009856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.011363 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.011366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.012026 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.012027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.012238 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.012853 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:17.012864 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:17.014784 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:17.016600 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.016605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.016607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.016833 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.017052 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.017052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.017053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.017155 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.017415 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:17.017419 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:17.017546 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.017629 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.017754 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":120,"threads_started":1}
I20260924 17:43:17.018579 2377 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39371
I20260924 17:43:17.018684 2377 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:17.018885 2377 heartbeater.cc:511] Master 127.1.8.190:39371 requested a full tablet report, sending...
I20260924 17:43:17.019475 2171 ts_manager.cc:194] Registered new tserver with Master: 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449)
I20260924 17:43:17.020175 2171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:49543
I20260924 17:43:17.022859 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 2246
I20260924 17:43:17.022967 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:17.023149 1058 external_mini_cluster.cc:1428] Setting key dae04feabd8b3e673d58b2b7aec0742d
I20260924 17:43:17.025841 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.019860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:48205 (local address 127.1.8.190:39371)
0924 17:43:17.019909 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.019911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.019974 (+ 63us) server_negotiation.cc:407] Connection header received
0924 17:43:17.020047 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.020048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.020059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.020106 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.020493 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.021009 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.021711 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.021916 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.021990 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.022981 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.022982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.022983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.022990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.024753 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.025083 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.025084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.025085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.025155 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.025389 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.025389 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.025389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.025493 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.025545 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.025705 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.025772 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:17.025941 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.019785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.1.8.129:48205)
0924 17:43:17.019920 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.019930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.019990 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.020108 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.020109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.020347 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:17.020415 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.020417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.021101 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.021102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.021624 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.021626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.021987 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.021988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.022066 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.022578 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:17.022582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:17.022882 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:17.024840 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.024841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.024842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.025009 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.025229 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.025230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.025230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.025310 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.025554 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:17.025555 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:17.025609 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.025683 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.025805 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:17.054128 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39371
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:17.137135 2386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:17.137369 2386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:17.137398 2386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:17.138652 2386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:17.138695 2386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:17.138715 2386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:43:17.138733 2386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:17.140163 2386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:17.140358 2386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:17.140545 2386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:17.145875 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:17.145875 2392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:17.145952 2394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:17.146071 2386 server_base.cc:1034] running on GCE node
I20260924 17:43:17.146450 2386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:17.146735 2386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:17.147949 2386 hybrid_clock.cc:648] HybridClock initialized: now 1790271797147927 us; error 32 us; skew 500 ppm
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:17.151050 2386 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:17.151587 2386 webserver.cc:460] Webserver started at http://127.1.8.130:39383/ using document root <none> and password file <none>
I20260924 17:43:17.151850 2386 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:17.151926 2386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:17.152091 2386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:17.153264 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "8b5cb676f2054fe8bdd68fc585977447"
format_stamp: "Formatted at 2026-09-24 17:43:17 on dist-test-slave-r7hf"
server_key: "9ed5955b0b8503dbd3488faf6c946a4b"
server_key_iv: "fb03703519f7d40bd6a4367ed942e23a"
server_key_version: "encryptionkey@0"
I20260924 17:43:17.153682 2386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "8b5cb676f2054fe8bdd68fc585977447"
format_stamp: "Formatted at 2026-09-24 17:43:17 on dist-test-slave-r7hf"
server_key: "9ed5955b0b8503dbd3488faf6c946a4b"
server_key_iv: "fb03703519f7d40bd6a4367ed942e23a"
server_key_version: "encryptionkey@0"
I20260924 17:43:17.155277 2386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:17.156214 2401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.156418 2386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:17.156538 2386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "8b5cb676f2054fe8bdd68fc585977447"
format_stamp: "Formatted at 2026-09-24 17:43:17 on dist-test-slave-r7hf"
server_key: "9ed5955b0b8503dbd3488faf6c946a4b"
server_key_iv: "fb03703519f7d40bd6a4367ed942e23a"
server_key_version: "encryptionkey@0"
I20260924 17:43:17.156622 2386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:17.175911 2386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:17.178313 2386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:17.178550 2386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:17.179143 2386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:17.179216 2386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.179273 2386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:17.179302 2386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.186975 2386 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:45521
I20260924 17:43:17.187083 2516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:45521 every 8 connection(s)
I20260924 17:43:17.187464 2386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:17.189529 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 2386
I20260924 17:43:17.189623 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:17.189774 1058 external_mini_cluster.cc:1428] Setting key b4ffbf7121af29f1f962a58546be4061
I20260924 17:43:17.191929 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.179901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:53649 (local address 127.1.8.190:39371)
0924 17:43:17.179960 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.179962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.180216 (+ 254us) server_negotiation.cc:407] Connection header received
0924 17:43:17.180804 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.180805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.180820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.180880 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.181921 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.182931 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.184713 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.185148 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.185236 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.188536 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.188538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.188538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.188548 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.190346 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.190860 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.190862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.190862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.190932 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.191172 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.191173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.191174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.191340 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.191417 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.191672 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.191771 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:17.191948 2410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.179841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.1.8.130:53649)
0924 17:43:17.180062 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.180102 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.180679 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.180972 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.180977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.181463 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:17.181776 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.181783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.183053 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.183054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.184559 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.184562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.185260 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.185261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.185413 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.186230 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:17.186244 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:17.188409 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:17.190502 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.190510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.190512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.190775 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.191005 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.191006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.191006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.191107 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.191409 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:17.191413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:17.191537 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.191621 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.191750 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260924 17:43:17.193584 2517 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39371
I20260924 17:43:17.193674 2517 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:17.193856 2517 heartbeater.cc:511] Master 127.1.8.190:39371 requested a full tablet report, sending...
I20260924 17:43:17.194271 2171 ts_manager.cc:194] Registered new tserver with Master: 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521)
I20260924 17:43:17.194918 2171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:53649
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:43:17.200822 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.193951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:52757 (local address 127.1.8.190:39371)
0924 17:43:17.193998 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.194000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.194262 (+ 262us) server_negotiation.cc:407] Connection header received
0924 17:43:17.194310 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.194311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.194329 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.194384 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.194943 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.195765 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.196508 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.196823 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.196902 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.198020 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.198021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.198021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.198029 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.199661 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.200016 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.200017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.200018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.200066 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.200352 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.200353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.200353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.200501 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.200555 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.200693 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.200754 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:17.201009 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.194019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.1.8.130:52757)
0924 17:43:17.194165 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.194177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.194252 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.194448 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.194449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.194769 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:17.194848 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.194851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.195876 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.195877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.196423 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.196424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.196933 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.196934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.197000 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.197575 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:17.197580 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:17.197924 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:17.199746 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.199747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.199747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.199948 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.200135 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.200136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.200136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.200246 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.200565 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:17.200565 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:17.200608 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.200740 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.200836 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:17.218081 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39371
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:17.304489 2526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:17.304718 2526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:17.304745 2526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:17.306015 2526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:17.306066 2526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:17.306088 2526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:43:17.306108 2526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:17.307742 2526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:39371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:17.308040 2526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:17.308302 2526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:17.313830 2532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:17.313844 2531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:17.313956 2534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:17.313890 2526 server_base.cc:1034] running on GCE node
I20260924 17:43:17.314285 2526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:17.314599 2526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:17.315800 2526 hybrid_clock.cc:648] HybridClock initialized: now 1790271797315828 us; error 78 us; skew 500 ppm
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:17.318770 2526 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:17.319396 2526 webserver.cc:460] Webserver started at http://127.1.8.131:44113/ using document root <none> and password file <none>
I20260924 17:43:17.319700 2526 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:17.319770 2526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:17.319919 2526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:17.321050 2526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "e62fb8ad07374b3fac7778c9dc0e8048"
format_stamp: "Formatted at 2026-09-24 17:43:17 on dist-test-slave-r7hf"
server_key: "fc8b81650cecba6ea3b7376f9829d4d0"
server_key_iv: "1ab311b45bff9b68b696e1e9446bf3de"
server_key_version: "encryptionkey@0"
I20260924 17:43:17.321432 2526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "e62fb8ad07374b3fac7778c9dc0e8048"
format_stamp: "Formatted at 2026-09-24 17:43:17 on dist-test-slave-r7hf"
server_key: "fc8b81650cecba6ea3b7376f9829d4d0"
server_key_iv: "1ab311b45bff9b68b696e1e9446bf3de"
server_key_version: "encryptionkey@0"
I20260924 17:43:17.323001 2526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 17:43:17.323885 2541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.324102 2526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:17.324196 2526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "e62fb8ad07374b3fac7778c9dc0e8048"
format_stamp: "Formatted at 2026-09-24 17:43:17 on dist-test-slave-r7hf"
server_key: "fc8b81650cecba6ea3b7376f9829d4d0"
server_key_iv: "1ab311b45bff9b68b696e1e9446bf3de"
server_key_version: "encryptionkey@0"
I20260924 17:43:17.324265 2526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:17.339624 2526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:17.341707 2526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:17.341830 2526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:17.342514 2526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:17.342559 2526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.342583 2526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:17.342595 2526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.349722 2526 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:41123
I20260924 17:43:17.349823 2656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:41123 every 8 connection(s)
I20260924 17:43:17.350176 2526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:17.353758 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 2526
I20260924 17:43:17.353886 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:17.354059 1058 external_mini_cluster.cc:1428] Setting key d6a1ab4f26c69044899d1d45b203fefa
I20260924 17:43:17.354636 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.343010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:54055 (local address 127.1.8.190:39371)
0924 17:43:17.343051 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.343053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.343329 (+ 276us) server_negotiation.cc:407] Connection header received
0924 17:43:17.343945 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.343946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.343960 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.344010 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.344993 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.345889 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.347624 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.347956 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.348056 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.351069 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.351071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.351071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.351081 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.352931 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.353387 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.353388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.353388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.353453 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.353811 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.353812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.353812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.353989 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.354062 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.354455 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.354545 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:17.354871 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.343011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.1.8.131:54055)
0924 17:43:17.343182 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.343240 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.343817 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.344089 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.344093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.344532 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:17.344857 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.344864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.346007 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.346009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.347501 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.347505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.348055 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.348055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.348165 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.348905 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:17.348918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:17.350951 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:17.353025 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.353033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.353035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.353282 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.353526 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.353526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.353526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.353630 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.354061 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:17.354069 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:17.354279 (+ 210us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.354397 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.354595 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260924 17:43:17.355392 2657 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39371
I20260924 17:43:17.355479 2657 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:17.355697 2657 heartbeater.cc:511] Master 127.1.8.190:39371 requested a full tablet report, sending...
I20260924 17:43:17.356112 2172 ts_manager.cc:194] Registered new tserver with Master: e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131:41123)
I20260924 17:43:17.356609 2172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:54055
I20260924 17:43:17.362823 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.356603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:51971 (local address 127.1.8.190:39371)
0924 17:43:17.356667 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.356669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.356795 (+ 126us) server_negotiation.cc:407] Connection header received
0924 17:43:17.356888 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.356889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.356902 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.356972 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.357351 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.357855 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.358846 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.359026 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.359084 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.360358 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.360360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.360360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.360367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.361768 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.362090 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.362090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.362091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.362140 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.362337 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.362338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.362338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.362509 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.362562 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.362694 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.362753 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:43:17.362824 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.356541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.1.8.131:51971)
0924 17:43:17.356730 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.356741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.356821 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.356992 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.356993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.357206 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:17.357268 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.357271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.357979 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.357980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.358764 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.358766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.359101 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.359102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.359166 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.359783 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:17.359789 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:17.360261 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:17.361845 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.361846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.361846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.362023 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.362212 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:17.362213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:17.362213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:17.362266 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:17.362555 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:17.362556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:17.362594 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.362650 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.362736 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260924 17:43:17.366145 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:43:17 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:17.401592 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.392967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.1.8.190:39371)
0924 17:43:17.393056 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.393058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.393069 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:17.393125 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.393126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.393141 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.393262 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:17.394194 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.395127 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.396119 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.396327 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.396390 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.398919 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:17.398920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:17.398921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:17.398929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:17.400465 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.400815 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.400816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.400817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.400873 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:17.401049 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:17.401050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:17.401050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:17.401262 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:17.401340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.401457 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.401521 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:43:17.403455 2172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:17.405218 2172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:17.412168 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.408135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.1.8.131:41123)
0924 17:43:17.408336 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.408340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.408352 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:43:17.408392 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.408393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.408406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.408500 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.408683 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.409646 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.410902 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.411664 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.411729 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.411797 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.412025 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.412067 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260924 17:43:17.412161 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.407868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:41123 (local address 127.0.0.1:44712)
0924 17:43:17.408049 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.408094 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.408166 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.408502 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.408506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.408508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.408583 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.408586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.409740 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.409741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.410822 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.410825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.411824 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.411825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.411892 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.411903 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.412004 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.412062 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
I20260924 17:43:17.412376 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.407873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:45521 (local address 127.0.0.1:60828)
0924 17:43:17.408742 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.408751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.408795 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.408966 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.408966 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.408968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.409019 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.409021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.410340 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.410342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.411248 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.411250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.412102 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.412103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.412166 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.412170 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.412219 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.412275 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":59,"threads_started":1}
I20260924 17:43:17.412714 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.408052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.1.8.130:45521)
0924 17:43:17.408272 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.408278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.408817 (+ 539us) server_negotiation.cc:407] Connection header received
0924 17:43:17.408866 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.408869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.408885 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.408971 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.409104 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.410244 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.411341 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.411947 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.412017 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.412069 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.412572 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.412616 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
I20260924 17:43:17.412930 2591 tablet_service.cc:1467] Processing CreateTablet for tablet 01d19f2c3e394ea9bada3cdedaec4ddb (DEFAULT_TABLE table=test-table [id=fb6cced355754717a54692a3a24f53ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:17.413339 2591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d19f2c3e394ea9bada3cdedaec4ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:17.413370 2451 tablet_service.cc:1467] Processing CreateTablet for tablet 01d19f2c3e394ea9bada3cdedaec4ddb (DEFAULT_TABLE table=test-table [id=fb6cced355754717a54692a3a24f53ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:17.413542 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.407868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42449 (local address 127.0.0.1:39788)
0924 17:43:17.408611 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.408621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.408664 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.409071 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.409072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.409075 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.409117 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.409119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.410367 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.410368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.411615 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.411618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.413310 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.413311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.413381 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.413385 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.413439 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.413480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260924 17:43:17.413763 2451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d19f2c3e394ea9bada3cdedaec4ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:17.413829 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.408147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.1.8.129:42449)
0924 17:43:17.408910 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.408915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.408923 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:17.408963 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.408964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.408979 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.409058 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.409192 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.410280 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.411713 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.412728 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.412899 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.412962 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.413690 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.413742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":80,"threads_started":1}
I20260924 17:43:17.415068 2311 tablet_service.cc:1467] Processing CreateTablet for tablet 01d19f2c3e394ea9bada3cdedaec4ddb (DEFAULT_TABLE table=test-table [id=fb6cced355754717a54692a3a24f53ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:17.415396 2311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d19f2c3e394ea9bada3cdedaec4ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:17.416733 2680 tablet_bootstrap.cc:492] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Bootstrap starting.
I20260924 17:43:17.418207 2680 tablet_bootstrap.cc:654] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:17.418757 2680 log.cc:824] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:17.418860 2681 tablet_bootstrap.cc:492] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Bootstrap starting.
I20260924 17:43:17.419655 2681 tablet_bootstrap.cc:654] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:17.419659 2682 tablet_bootstrap.cc:492] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Bootstrap starting.
I20260924 17:43:17.419996 2681 log.cc:824] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:17.420315 2682 tablet_bootstrap.cc:654] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:17.420938 2681 tablet_bootstrap.cc:492] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: No bootstrap required, opened a new log
I20260924 17:43:17.421016 2681 ts_tablet_manager.cc:1397] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:43:17.421823 2680 tablet_bootstrap.cc:492] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: No bootstrap required, opened a new log
I20260924 17:43:17.421903 2680 ts_tablet_manager.cc:1397] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 17:43:17.422600 2682 log.cc:824] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:17.423458 2681 raft_consensus.cc:348] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.423661 2681 raft_consensus.cc:374] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:17.423722 2681 raft_consensus.cc:729] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5cb676f2054fe8bdd68fc585977447, State: Initialized, Role: FOLLOWER
I20260924 17:43:17.423861 2681 consensus_queue.cc:260] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.424111 2682 tablet_bootstrap.cc:492] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: No bootstrap required, opened a new log
I20260924 17:43:17.424139 2681 ts_tablet_manager.cc:1428] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:17.424225 2682 ts_tablet_manager.cc:1397] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 17:43:17.424283 2517 heartbeater.cc:503] Master 127.1.8.190:39371 was elected leader, sending a full tablet report...
I20260924 17:43:17.425693 2680 raft_consensus.cc:348] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.425879 2680 raft_consensus.cc:374] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:17.425918 2680 raft_consensus.cc:729] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fb8ad07374b3fac7778c9dc0e8048, State: Initialized, Role: FOLLOWER
I20260924 17:43:17.426065 2680 consensus_queue.cc:260] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.426235 2682 raft_consensus.cc:348] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.426488 2682 raft_consensus.cc:374] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:17.426543 2682 raft_consensus.cc:729] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44727c74c5944ebe8429b0e77c79304e, State: Initialized, Role: FOLLOWER
I20260924 17:43:17.426458 2680 ts_tablet_manager.cc:1428] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 17:43:17.426666 2657 heartbeater.cc:503] Master 127.1.8.190:39371 was elected leader, sending a full tablet report...
I20260924 17:43:17.426671 2682 consensus_queue.cc:260] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.426924 2682 ts_tablet_manager.cc:1428] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:17.426994 2377 heartbeater.cc:503] Master 127.1.8.190:39371 was elected leader, sending a full tablet report...
W20260924 17:43:17.438155 2518 tablet.cc:2369] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:43:17.514505 2378 tablet.cc:2369] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:43:17.600898 2658 tablet.cc:2369] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:43:17.635772 2687 raft_consensus.cc:482] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:17.636030 2687 raft_consensus.cc:504] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.636492 2687 leader_election.cc:288] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449), 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521)
I20260924 17:43:17.640002 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.636726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42449 (local address 127.1.8.131:39307)
0924 17:43:17.636788 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.636804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.636851 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.637052 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.637053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.637056 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.637139 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.637141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.637859 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.637860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.638911 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.638913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.639691 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.639692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.639803 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.639808 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.639872 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.639914 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":2}
I20260924 17:43:17.640033 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.636775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39307 (local address 127.1.8.129:42449)
0924 17:43:17.636827 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.636829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.636842 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:43:17.636958 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.636959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.636974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.637028 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.637228 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.637748 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.639029 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.639608 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.639685 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.639698 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.639893 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.639959 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:17.640105 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.636742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:45521 (local address 127.1.8.131:36641)
0924 17:43:17.636987 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.636999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.637080 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.637221 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.637222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.637224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.637300 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.637302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.637936 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.637937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.639024 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.639027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.639733 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.639734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.639812 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.639814 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.639953 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.639987 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":3,"thread_start_us":160,"threads_started":1}
I20260924 17:43:17.640107 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.636902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:36641 (local address 127.1.8.130:45521)
0924 17:43:17.636952 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.636955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.637057 (+ 102us) server_negotiation.cc:407] Connection header received
0924 17:43:17.637150 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.637151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.637163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.637214 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.637381 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.637848 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.639111 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.639665 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.639726 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.639735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.639981 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.640023 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:43:17.640384 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d19f2c3e394ea9bada3cdedaec4ddb" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44727c74c5944ebe8429b0e77c79304e" is_pre_election: true
I20260924 17:43:17.640522 2331 raft_consensus.cc:2454] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 0.
I20260924 17:43:17.640743 2471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d19f2c3e394ea9bada3cdedaec4ddb" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5cb676f2054fe8bdd68fc585977447" is_pre_election: true
I20260924 17:43:17.640758 2545 leader_election.cc:302] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44727c74c5944ebe8429b0e77c79304e, e62fb8ad07374b3fac7778c9dc0e8048; no voters:
I20260924 17:43:17.640856 2471 raft_consensus.cc:2454] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 0.
I20260924 17:43:17.640990 2687 raft_consensus.cc:2781] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:17.641062 2687 raft_consensus.cc:482] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:17.641096 2687 raft_consensus.cc:3037] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:17.642074 2687 raft_consensus.cc:504] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.642232 2687 leader_election.cc:288] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 election: Requested vote from peers 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449), 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521)
I20260924 17:43:17.642372 2471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d19f2c3e394ea9bada3cdedaec4ddb" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5cb676f2054fe8bdd68fc585977447"
I20260924 17:43:17.642431 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d19f2c3e394ea9bada3cdedaec4ddb" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44727c74c5944ebe8429b0e77c79304e"
I20260924 17:43:17.642489 2471 raft_consensus.cc:3037] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:17.642500 2331 raft_consensus.cc:3037] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:17.643409 2471 raft_consensus.cc:2454] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 1.
I20260924 17:43:17.643409 2331 raft_consensus.cc:2454] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 1.
I20260924 17:43:17.643587 2543 leader_election.cc:302] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5cb676f2054fe8bdd68fc585977447, e62fb8ad07374b3fac7778c9dc0e8048; no voters:
I20260924 17:43:17.643695 2687 raft_consensus.cc:2781] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:17.643831 2687 raft_consensus.cc:686] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 LEADER]: Becoming Leader. State: Replica: e62fb8ad07374b3fac7778c9dc0e8048, State: Running, Role: LEADER
I20260924 17:43:17.643996 2687 consensus_queue.cc:237] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } }
I20260924 17:43:17.644893 2171 catalog_manager.cc:5495] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 reported cstate change: term changed from 0 to 1, leader changed from <none> to e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131). New cstate: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } health_report { overall_health: UNKNOWN } } }
I20260924 17:43:17.699329 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.696600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.1.8.131:41123)
0924 17:43:17.696675 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.696678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.696687 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:17.696766 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.696767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.696794 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.696849 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:17.697093 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.697581 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.698422 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.698608 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.698684 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.698944 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:17.699051 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:17.699141 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.699182 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.699206 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:43:17.701351 2471 raft_consensus.cc:1261] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Refusing update from remote peer e62fb8ad07374b3fac7778c9dc0e8048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:17.701376 2331 raft_consensus.cc:1261] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Refusing update from remote peer e62fb8ad07374b3fac7778c9dc0e8048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:17.701710 2687 consensus_queue.cc:1035] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:17.701838 2690 consensus_queue.cc:1035] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:17.704552 2693 mvcc.cc:205] Tried to move back new op lower bound from 7332953283381723136 to 7332953283150606336. Current Snapshot: MvccSnapshot[applied={T|T < 7332953283381723136}]
I20260924 17:43:17.708588 2172 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:17.708716 2172 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:17.709614 2172 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: Sending DeleteTablet for 3 replicas of tablet 01d19f2c3e394ea9bada3cdedaec4ddb
I20260924 17:43:17.709969 2451 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d19f2c3e394ea9bada3cdedaec4ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:17 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:60828
I20260924 17:43:17.710004 2591 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d19f2c3e394ea9bada3cdedaec4ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:17 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:44712
I20260924 17:43:17.710248 2703 tablet_replica.cc:330] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: stopping tablet replica
I20260924 17:43:17.710489 2311 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d19f2c3e394ea9bada3cdedaec4ddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:17 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39788
I20260924 17:43:17.710678 2703 raft_consensus.cc:2229] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:17.710728 2172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 17:43:17.710814 2703 raft_consensus.cc:2258] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 17:43:17.710880 2172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:17.711062 2704 tablet_replica.cc:330] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: stopping tablet replica
I20260924 17:43:17.711158 2704 raft_consensus.cc:2229] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:17.711278 2704 raft_consensus.cc:2258] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:17.711663 2703 ts_tablet_manager.cc:1905] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:17.712240 2451 tablet_service.cc:1467] Processing CreateTablet for tablet 1a544713abbe47c89940e117528a5753 (DEFAULT_TABLE table=test-table [id=cd6291e13bc14bc693ba676fee33f3f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:17.712287 2311 tablet_service.cc:1467] Processing CreateTablet for tablet 1a544713abbe47c89940e117528a5753 (DEFAULT_TABLE table=test-table [id=cd6291e13bc14bc693ba676fee33f3f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:17.712373 2451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a544713abbe47c89940e117528a5753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:17.712423 2311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a544713abbe47c89940e117528a5753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:17.713424 2682 tablet_bootstrap.cc:492] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Bootstrap starting.
I20260924 17:43:17.713935 2682 tablet_bootstrap.cc:654] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:17.714083 2681 tablet_bootstrap.cc:492] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Bootstrap starting.
I20260924 17:43:17.714251 2591 tablet_service.cc:1467] Processing CreateTablet for tablet 1a544713abbe47c89940e117528a5753 (DEFAULT_TABLE table=test-table [id=cd6291e13bc14bc693ba676fee33f3f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:17.714376 2591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a544713abbe47c89940e117528a5753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:17.714514 2681 tablet_bootstrap.cc:654] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:17.715232 2681 tablet_bootstrap.cc:492] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: No bootstrap required, opened a new log
I20260924 17:43:17.715293 2681 ts_tablet_manager.cc:1397] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:17.715438 2681 raft_consensus.cc:348] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.715508 2681 raft_consensus.cc:374] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:17.715528 2681 raft_consensus.cc:729] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5cb676f2054fe8bdd68fc585977447, State: Initialized, Role: FOLLOWER
I20260924 17:43:17.715579 2681 consensus_queue.cc:260] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.715700 2681 ts_tablet_manager.cc:1428] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.715735 2680 tablet_bootstrap.cc:492] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Bootstrap starting.
I20260924 17:43:17.716132 2704 ts_tablet_manager.cc:1905] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:17.716181 2680 tablet_bootstrap.cc:654] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:17.716853 2680 tablet_bootstrap.cc:492] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: No bootstrap required, opened a new log
I20260924 17:43:17.716904 2680 ts_tablet_manager.cc:1397] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:17.717056 2680 raft_consensus.cc:348] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.717128 2680 raft_consensus.cc:374] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:17.717149 2680 raft_consensus.cc:729] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fb8ad07374b3fac7778c9dc0e8048, State: Initialized, Role: FOLLOWER
I20260924 17:43:17.717201 2680 consensus_queue.cc:260] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.717283 2680 ts_tablet_manager.cc:1428] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.718015 2705 tablet_replica.cc:330] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: stopping tablet replica
I20260924 17:43:17.718027 2704 ts_tablet_manager.cc:1918] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:17.718097 2704 log.cc:1196] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/01d19f2c3e394ea9bada3cdedaec4ddb
I20260924 17:43:17.718112 2682 tablet_bootstrap.cc:492] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: No bootstrap required, opened a new log
I20260924 17:43:17.718150 2682 ts_tablet_manager.cc:1397] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 17:43:17.718298 2703 ts_tablet_manager.cc:1918] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:17.718441 2704 ts_tablet_manager.cc:1939] T 01d19f2c3e394ea9bada3cdedaec4ddb P e62fb8ad07374b3fac7778c9dc0e8048: Deleting consensus metadata
I20260924 17:43:17.718402 2682 raft_consensus.cc:348] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.718453 2703 log.cc:1196] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/01d19f2c3e394ea9bada3cdedaec4ddb
I20260924 17:43:17.718480 2682 raft_consensus.cc:374] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:17.718511 2682 raft_consensus.cc:729] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44727c74c5944ebe8429b0e77c79304e, State: Initialized, Role: FOLLOWER
I20260924 17:43:17.718552 2682 consensus_queue.cc:260] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.718663 2682 ts_tablet_manager.cc:1428] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:17.718143 2705 raft_consensus.cc:2229] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:17.718979 2160 catalog_manager.cc:4841] TS e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131:41123): tablet 01d19f2c3e394ea9bada3cdedaec4ddb (table test-table [id=fb6cced355754717a54692a3a24f53ea]) successfully deleted
I20260924 17:43:17.719075 2703 ts_tablet_manager.cc:1939] T 01d19f2c3e394ea9bada3cdedaec4ddb P 8b5cb676f2054fe8bdd68fc585977447: Deleting consensus metadata
I20260924 17:43:17.719245 2705 raft_consensus.cc:2258] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:17.719677 2705 ts_tablet_manager.cc:1905] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:17.721172 2705 ts_tablet_manager.cc:1918] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:17.721232 2705 log.cc:1196] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/01d19f2c3e394ea9bada3cdedaec4ddb
I20260924 17:43:17.721927 2705 ts_tablet_manager.cc:1939] T 01d19f2c3e394ea9bada3cdedaec4ddb P 44727c74c5944ebe8429b0e77c79304e: Deleting consensus metadata
I20260924 17:43:17.722364 2161 catalog_manager.cc:4841] TS 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449): tablet 01d19f2c3e394ea9bada3cdedaec4ddb (table test-table [id=fb6cced355754717a54692a3a24f53ea]) successfully deleted
I20260924 17:43:17.722620 2159 catalog_manager.cc:4841] TS 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521): tablet 01d19f2c3e394ea9bada3cdedaec4ddb (table test-table [id=fb6cced355754717a54692a3a24f53ea]) successfully deleted
I20260924 17:43:17.979888 2686 raft_consensus.cc:482] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:17.980008 2686 raft_consensus.cc:504] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.980420 2686 leader_election.cc:288] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449), e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131:41123)
I20260924 17:43:17.983846 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.980641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42449 (local address 127.1.8.130:32815)
0924 17:43:17.980830 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.980839 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.980908 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.981087 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.981088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.981092 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.981210 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.981213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.981912 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.981914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.983025 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.983028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.983626 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.983627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.983676 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.983679 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.983721 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.983752 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":7,"thread_start_us":64,"threads_started":1}
I20260924 17:43:17.983871 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.980735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:32815 (local address 127.1.8.129:42449)
0924 17:43:17.980787 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.980790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.980918 (+ 128us) server_negotiation.cc:407] Connection header received
0924 17:43:17.980971 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.980972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.980982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.981067 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.981290 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.981814 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.983145 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.983524 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.983610 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.983623 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.983751 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.983796 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:17.984149 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a544713abbe47c89940e117528a5753" candidate_uuid: "8b5cb676f2054fe8bdd68fc585977447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44727c74c5944ebe8429b0e77c79304e" is_pre_election: true
I20260924 17:43:17.984259 2331 raft_consensus.cc:2454] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5cb676f2054fe8bdd68fc585977447 in term 0.
I20260924 17:43:17.984498 2405 leader_election.cc:302] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44727c74c5944ebe8429b0e77c79304e, 8b5cb676f2054fe8bdd68fc585977447; no voters:
I20260924 17:43:17.984616 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.980641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:41123 (local address 127.1.8.130:37907)
0924 17:43:17.980821 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:17.980833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:17.980908 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:17.981087 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:17.981089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:17.981091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:17.981210 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.981213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.982617 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.982618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.983465 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:17.983467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.984324 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.984325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:17.984432 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.984437 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:17.984498 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 17:43:17.984540 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":58,"thread_start_us":78,"threads_started":1}
I20260924 17:43:17.984650 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:17.980735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:37907 (local address 127.1.8.131:41123)
0924 17:43:17.980787 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:17.980790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:17.980918 (+ 128us) server_negotiation.cc:407] Connection header received
0924 17:43:17.980963 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:17.980965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:17.980976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:17.981067 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:17.981930 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.982538 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.983566 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:17.984230 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:17.984314 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:17.984325 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:17.984517 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 17:43:17.984554 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:17.984647 2686 raft_consensus.cc:2781] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:17.984700 2686 raft_consensus.cc:482] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:17.984724 2686 raft_consensus.cc:3037] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:17.985492 2611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a544713abbe47c89940e117528a5753" candidate_uuid: "8b5cb676f2054fe8bdd68fc585977447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" is_pre_election: true
I20260924 17:43:17.985641 2611 raft_consensus.cc:2454] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5cb676f2054fe8bdd68fc585977447 in term 0.
I20260924 17:43:17.986164 2686 raft_consensus.cc:504] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.986322 2686 leader_election.cc:288] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [CANDIDATE]: Term 1 election: Requested vote from peers 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449), e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131:41123)
I20260924 17:43:17.986527 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a544713abbe47c89940e117528a5753" candidate_uuid: "8b5cb676f2054fe8bdd68fc585977447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44727c74c5944ebe8429b0e77c79304e"
I20260924 17:43:17.986511 2611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a544713abbe47c89940e117528a5753" candidate_uuid: "8b5cb676f2054fe8bdd68fc585977447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e62fb8ad07374b3fac7778c9dc0e8048"
I20260924 17:43:17.986599 2331 raft_consensus.cc:3037] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:17.986622 2611 raft_consensus.cc:3037] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:17.987290 2611 raft_consensus.cc:2454] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5cb676f2054fe8bdd68fc585977447 in term 1.
I20260924 17:43:17.987290 2331 raft_consensus.cc:2454] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5cb676f2054fe8bdd68fc585977447 in term 1.
I20260924 17:43:17.987519 2405 leader_election.cc:302] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44727c74c5944ebe8429b0e77c79304e, 8b5cb676f2054fe8bdd68fc585977447; no voters:
I20260924 17:43:17.987649 2686 raft_consensus.cc:2781] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:17.987812 2686 raft_consensus.cc:686] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 LEADER]: Becoming Leader. State: Replica: 8b5cb676f2054fe8bdd68fc585977447, State: Running, Role: LEADER
I20260924 17:43:17.987901 2686 consensus_queue.cc:237] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:17.988673 2172 catalog_manager.cc:5495] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "8b5cb676f2054fe8bdd68fc585977447" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } health_report { overall_health: UNKNOWN } } }
Sep 24 17:43:18 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271796, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:18.035152 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.027325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39371 (local address 127.0.0.1:50066)
0924 17:43:18.027515 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:18.027526 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:18.027631 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:18.027886 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:18.027886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:18.028196 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:18.028240 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.028242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.028921 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.028922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.029455 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.029456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.029859 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.029860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.029919 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.030521 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:18.030526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:18.032194 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:18.033981 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:18.033990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:18.033992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:18.034216 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:18.034568 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:18.034569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:18.034569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:18.034634 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:18.034875 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:18.034881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:18.034928 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 17:43:18.034983 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:43:18.035074 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260924 17:43:18.035298 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.027371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.1.8.190:39371)
0924 17:43:18.027512 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.027514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.027609 (+ 95us) server_negotiation.cc:407] Connection header received
0924 17:43:18.027707 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.027708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.027721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.027821 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:18.028334 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.028839 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.029554 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.029748 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.029846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.032328 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:18.032329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:18.032330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:18.032337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:18.033888 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:18.034288 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:18.034289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:18.034290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:18.034359 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:18.034689 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:18.034690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:18.034691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:18.034807 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:18.034875 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.035062 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.035137 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260924 17:43:18.037695 2172 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:50066:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 17:43:18.037962 2172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:18.039361 2451 tablet_service.cc:1467] Processing CreateTablet for tablet 66e9ce774f664433b470a8f566acff98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0e49d94fbe44da3b95a3c60ca4c52f8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:43:18.039359 2591 tablet_service.cc:1467] Processing CreateTablet for tablet 66e9ce774f664433b470a8f566acff98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0e49d94fbe44da3b95a3c60ca4c52f8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:43:18.039390 2311 tablet_service.cc:1467] Processing CreateTablet for tablet 66e9ce774f664433b470a8f566acff98 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a0e49d94fbe44da3b95a3c60ca4c52f8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:43:18.039528 2591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e9ce774f664433b470a8f566acff98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:18.039528 2451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e9ce774f664433b470a8f566acff98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:18.039547 2311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66e9ce774f664433b470a8f566acff98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:18.040768 2682 tablet_bootstrap.cc:492] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: Bootstrap starting.
I20260924 17:43:18.041139 2682 tablet_bootstrap.cc:654] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:18.041420 2680 tablet_bootstrap.cc:492] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: Bootstrap starting.
I20260924 17:43:18.041738 2680 tablet_bootstrap.cc:654] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:18.041919 2682 tablet_bootstrap.cc:492] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: No bootstrap required, opened a new log
I20260924 17:43:18.041962 2682 ts_tablet_manager.cc:1397] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:18.042055 2681 tablet_bootstrap.cc:492] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: Bootstrap starting.
I20260924 17:43:18.042115 2682 raft_consensus.cc:348] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.042196 2682 raft_consensus.cc:374] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:18.042220 2682 raft_consensus.cc:729] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44727c74c5944ebe8429b0e77c79304e, State: Initialized, Role: FOLLOWER
I20260924 17:43:18.042292 2682 consensus_queue.cc:260] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.042335 2680 tablet_bootstrap.cc:492] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: No bootstrap required, opened a new log
I20260924 17:43:18.042412 2682 ts_tablet_manager.cc:1428] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:18.042382 2680 ts_tablet_manager.cc:1397] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:18.042469 2688 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.042559 2688 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.042546 2680 raft_consensus.cc:348] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.042609 2680 raft_consensus.cc:374] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:18.042603 2681 tablet_bootstrap.cc:654] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:18.042631 2680 raft_consensus.cc:729] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62fb8ad07374b3fac7778c9dc0e8048, State: Initialized, Role: FOLLOWER
I20260924 17:43:18.042673 2680 consensus_queue.cc:260] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.042750 2680 ts_tablet_manager.cc:1428] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:18.042980 2690 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.043097 2690 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.043349 2681 tablet_bootstrap.cc:492] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: No bootstrap required, opened a new log
I20260924 17:43:18.043406 2681 ts_tablet_manager.cc:1397] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:18.043545 2681 raft_consensus.cc:348] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.043630 2681 raft_consensus.cc:374] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:18.043654 2681 raft_consensus.cc:729] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5cb676f2054fe8bdd68fc585977447, State: Initialized, Role: FOLLOWER
I20260924 17:43:18.043708 2681 consensus_queue.cc:260] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.043807 2681 ts_tablet_manager.cc:1428] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:18.043859 2686 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.043948 2686 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.145428 2690 raft_consensus.cc:482] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:18.145529 2690 raft_consensus.cc:504] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.145766 2690 leader_election.cc:288] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449), 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521)
I20260924 17:43:18.146047 2471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e9ce774f664433b470a8f566acff98" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5cb676f2054fe8bdd68fc585977447" is_pre_election: true
I20260924 17:43:18.146050 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e9ce774f664433b470a8f566acff98" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44727c74c5944ebe8429b0e77c79304e" is_pre_election: true
I20260924 17:43:18.146131 2471 raft_consensus.cc:2454] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 0.
I20260924 17:43:18.146167 2331 raft_consensus.cc:2454] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 0.
I20260924 17:43:18.146293 2543 leader_election.cc:302] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5cb676f2054fe8bdd68fc585977447, e62fb8ad07374b3fac7778c9dc0e8048; no voters:
I20260924 17:43:18.146382 2690 raft_consensus.cc:2781] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:18.146452 2690 raft_consensus.cc:482] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:18.146469 2690 raft_consensus.cc:3037] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:18.147161 2690 raft_consensus.cc:504] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.147344 2690 leader_election.cc:288] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 election: Requested vote from peers 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449), 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521)
I20260924 17:43:18.147500 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e9ce774f664433b470a8f566acff98" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44727c74c5944ebe8429b0e77c79304e"
I20260924 17:43:18.147526 2471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66e9ce774f664433b470a8f566acff98" candidate_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b5cb676f2054fe8bdd68fc585977447"
I20260924 17:43:18.147593 2471 raft_consensus.cc:3037] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:18.147593 2331 raft_consensus.cc:3037] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:18.148389 2471 raft_consensus.cc:2454] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 1.
I20260924 17:43:18.148389 2331 raft_consensus.cc:2454] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62fb8ad07374b3fac7778c9dc0e8048 in term 1.
I20260924 17:43:18.148578 2545 leader_election.cc:302] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44727c74c5944ebe8429b0e77c79304e, e62fb8ad07374b3fac7778c9dc0e8048; no voters:
I20260924 17:43:18.148684 2690 raft_consensus.cc:2781] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:18.148826 2690 raft_consensus.cc:686] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 LEADER]: Becoming Leader. State: Replica: e62fb8ad07374b3fac7778c9dc0e8048, State: Running, Role: LEADER
I20260924 17:43:18.148891 2690 consensus_queue.cc:237] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } }
I20260924 17:43:18.149199 2687 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: TxnStatusTablet state changed. Reason: New leader e62fb8ad07374b3fac7778c9dc0e8048. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.149303 2687 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:43:18.149443 2724 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 17:43:18.149564 2724 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 17:43:18.149533 2172 catalog_manager.cc:5495] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 reported cstate change: term changed from 0 to 1, leader changed from <none> to e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131). New cstate: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } health_report { overall_health: HEALTHY } } }
I20260924 17:43:18.163897 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.161488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.1.8.131:41123)
0924 17:43:18.161546 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.161548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.161557 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:18.161663 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.161664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.161679 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.161737 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:18.161861 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.162350 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.163175 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.163377 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.163452 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.163618 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.163655 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.163724 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.163785 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.163807 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:18.163899 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.161402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:41123 (local address 127.0.0.1:44728)
0924 17:43:18.161488 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:18.161499 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:18.161574 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:18.161743 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:18.161743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:18.161744 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:18.161799 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.161802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.162494 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.162495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.163079 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.163080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.163450 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.163451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.163511 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.163526 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.163725 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.163735 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 17:43:18.163781 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 17:43:18.163831 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 17:43:18.165127 2471 raft_consensus.cc:1261] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Refusing update from remote peer e62fb8ad07374b3fac7778c9dc0e8048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:18.165145 2331 raft_consensus.cc:1261] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Refusing update from remote peer e62fb8ad07374b3fac7778c9dc0e8048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:18.165377 2687 consensus_queue.cc:1035] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:18.165534 2690 consensus_queue.cc:1035] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:18.165940 2688 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: TxnStatusTablet state changed. Reason: New leader e62fb8ad07374b3fac7778c9dc0e8048. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.166028 2688 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.166160 2686 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: TxnStatusTablet state changed. Reason: New leader e62fb8ad07374b3fac7778c9dc0e8048. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.166237 2686 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.166950 2698 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.167038 2698 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:43:18.167178 2694 mvcc.cc:205] Tried to move back new op lower bound from 7332953285282545664 to 7332953285218181120. Current Snapshot: MvccSnapshot[applied={T|T < 7332953285282545664 or (T in {7332953285282545664})}]
I20260924 17:43:18.167488 2686 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.167565 2686 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P 8b5cb676f2054fe8bdd68fc585977447: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.168241 2687 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.168318 2687 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P e62fb8ad07374b3fac7778c9dc0e8048: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:43:18.168534 2688 tablet_replica.cc:439] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 } } peers { permanent_uuid: "8b5cb676f2054fe8bdd68fc585977447" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 45521 } } peers { permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 } } }
I20260924 17:43:18.168653 2688 tablet_replica.cc:442] T 66e9ce774f664433b470a8f566acff98 P 44727c74c5944ebe8429b0e77c79304e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:18.171859 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.169307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.1.8.130:45521)
0924 17:43:18.169463 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.169464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.169472 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:18.169503 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.169503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.169519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.169611 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:18.169723 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.170245 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.171010 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.171227 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.171309 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.171475 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.171590 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.171670 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.171723 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.171743 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260924 17:43:18.177323 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.174515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:42155 (local address 127.1.8.131:41123)
0924 17:43:18.174582 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.174584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.174689 (+ 105us) server_negotiation.cc:407] Connection header received
0924 17:43:18.174759 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.174760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.174772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.174829 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:18.174982 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.175440 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.176605 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.176816 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.176873 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.177087 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.177121 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.177172 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.177245 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.177265 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:43:18.177378 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.174478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:41123 (local address 127.1.8.130:42155)
0924 17:43:18.174626 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:18.174637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:18.174699 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:18.174846 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:18.174847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:18.174849 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:18.174912 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.174914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.175514 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.175515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.176521 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.176525 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.176888 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.176889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.176964 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.176983 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.177196 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.177198 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:43:18.177243 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 17:43:18.177289 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 17:43:18.180169 2611 raft_consensus.cc:1261] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Refusing update from remote peer 8b5cb676f2054fe8bdd68fc585977447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:18.180464 2331 raft_consensus.cc:1261] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Refusing update from remote peer 8b5cb676f2054fe8bdd68fc585977447: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:18.180574 2711 consensus_queue.cc:1035] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e62fb8ad07374b3fac7778c9dc0e8048" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 41123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:18.180866 2711 consensus_queue.cc:1035] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44727c74c5944ebe8429b0e77c79304e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:18.182993 2695 mvcc.cc:205] Tried to move back new op lower bound from 7332953285343444992 to 7332953284558794752. Current Snapshot: MvccSnapshot[applied={T|T < 7332953285343444992}]
I20260924 17:43:18.189764 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.186556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:36761 (local address 127.1.8.130:45521)
0924 17:43:18.186666 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.186668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.186745 (+ 77us) server_negotiation.cc:407] Connection header received
0924 17:43:18.186820 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.186821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.186830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.186878 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:18.187046 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.187774 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.188951 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.189255 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.189340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.189506 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.189542 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.189593 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.189665 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.189686 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 17:43:18.189821 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.186517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:45521 (local address 127.1.8.131:36761)
0924 17:43:18.186689 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:18.186702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:18.186763 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:18.186927 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:18.186928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:18.186929 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:18.186980 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.186981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.187923 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.187924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.188856 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:18.188859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.189347 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.189348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:18.189405 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.189423 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:18.189604 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:18.189606 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:43:18.189659 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:43:18.189707 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260924 17:43:18.198448 2171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:18.198585 2171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:18.199214 2171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19c8d985696a42529c8d70e121eb4446: Sending DeleteTablet for 3 replicas of tablet 1a544713abbe47c89940e117528a5753
I20260924 17:43:18.199493 2451 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a544713abbe47c89940e117528a5753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:18 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:60828
I20260924 17:43:18.199548 2591 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a544713abbe47c89940e117528a5753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:18 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:44712
I20260924 17:43:18.199633 2703 tablet_replica.cc:330] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: stopping tablet replica
I20260924 17:43:18.199682 2703 raft_consensus.cc:2229] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:18.199781 2703 raft_consensus.cc:2258] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:18.199903 2704 tablet_replica.cc:330] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: stopping tablet replica
I20260924 17:43:18.199967 2704 raft_consensus.cc:2229] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:18.200011 2704 raft_consensus.cc:2258] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:18.200358 2311 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a544713abbe47c89940e117528a5753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:18 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39788
I20260924 17:43:18.200467 2705 tablet_replica.cc:330] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: stopping tablet replica
I20260924 17:43:18.200533 2704 ts_tablet_manager.cc:1905] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:18.200552 2705 raft_consensus.cc:2229] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:18.200591 2705 raft_consensus.cc:2258] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:18.200929 2705 ts_tablet_manager.cc:1905] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:18.202294 2704 ts_tablet_manager.cc:1918] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:18.202371 2704 log.cc:1196] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/1a544713abbe47c89940e117528a5753
I20260924 17:43:18.202423 2705 ts_tablet_manager.cc:1918] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:18.202481 2705 log.cc:1196] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/1a544713abbe47c89940e117528a5753
I20260924 17:43:18.202636 2704 ts_tablet_manager.cc:1939] T 1a544713abbe47c89940e117528a5753 P e62fb8ad07374b3fac7778c9dc0e8048: Deleting consensus metadata
I20260924 17:43:18.202697 2705 ts_tablet_manager.cc:1939] T 1a544713abbe47c89940e117528a5753 P 44727c74c5944ebe8429b0e77c79304e: Deleting consensus metadata
I20260924 17:43:18.202734 2703 ts_tablet_manager.cc:1905] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:18.203006 2160 catalog_manager.cc:4841] TS e62fb8ad07374b3fac7778c9dc0e8048 (127.1.8.131:41123): tablet 1a544713abbe47c89940e117528a5753 (table test-table [id=cd6291e13bc14bc693ba676fee33f3f4]) successfully deleted
I20260924 17:43:18.203459 2161 catalog_manager.cc:4841] TS 44727c74c5944ebe8429b0e77c79304e (127.1.8.129:42449): tablet 1a544713abbe47c89940e117528a5753 (table test-table [id=cd6291e13bc14bc693ba676fee33f3f4]) successfully deleted
I20260924 17:43:18.204317 2703 ts_tablet_manager.cc:1918] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:18.204402 2703 log.cc:1196] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/1a544713abbe47c89940e117528a5753
I20260924 17:43:18.204608 2703 ts_tablet_manager.cc:1939] T 1a544713abbe47c89940e117528a5753 P 8b5cb676f2054fe8bdd68fc585977447: Deleting consensus metadata
I20260924 17:43:18.204924 2159 catalog_manager.cc:4841] TS 8b5cb676f2054fe8bdd68fc585977447 (127.1.8.130:45521): tablet 1a544713abbe47c89940e117528a5753 (table test-table [id=cd6291e13bc14bc693ba676fee33f3f4]) successfully deleted
Sep 24 17:43:18 dist-test-slave-r7hf krb5kdc[2129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:18.215384 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.206848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.1.8.129:42449)
0924 17:43:18.206898 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.206900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.206978 (+ 78us) server_negotiation.cc:407] Connection header received
0924 17:43:18.207063 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.207064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.207074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.207117 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:18.207698 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.208195 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.208955 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.209190 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.209280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.212229 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:18.212243 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:18.212243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:18.212251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:18.214005 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:18.214448 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:18.214452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:18.214453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:18.214526 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:18.214747 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:18.214748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:18.214748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:18.214908 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:18.215065 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.215134 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.215226 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:43:18.215658 2371 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39790
I20260924 17:43:18.223843 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:18.216719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.1.8.190:39371)
0924 17:43:18.216767 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:18.216769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:18.216828 (+ 59us) server_negotiation.cc:407] Connection header received
0924 17:43:18.216901 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:18.216902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:18.216913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:18.216958 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:18.217539 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.218208 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.219269 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:18.219491 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:18.219591 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:18.220922 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:18.220924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:18.220924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:18.220932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:18.222640 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:18.223019 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:18.223020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:18.223021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:18.223088 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:18.223296 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:18.223297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:18.223297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:18.223426 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:18.223536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:18.223640 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 17:43:18.223734 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:43:18.224205 2171 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50070
I20260924 17:43:18.224814 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 2246
I20260924 17:43:18.231217 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 2386
I20260924 17:43:18.238014 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 2526
I20260924 17:43:18.245215 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 2142
2026-09-24T17:43:18Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2764 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:18 dist-test-slave-r7hf krb5kdc[2746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:18 dist-test-slave-r7hf krb5kdc[2746](info): set up 2 sockets
Sep 24 17:43:18 dist-test-slave-r7hf krb5kdc[2746](info): commencing operation
krb5kdc: starting...
W20260924 17:43:19.294176 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:19.421955 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:34169
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:34169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:19.504130 2759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:19.504410 2759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:19.504451 2759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:19.505656 2759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:19.505733 2759 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 17:43:19.505755 2759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:19.505776 2759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:19.505805 2759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:19.505836 2759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:19.507376 2759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:34169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:19.507687 2759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:19.507931 2759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:19.513276 2764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:19.513337 2759 server_base.cc:1034] running on GCE node
W20260924 17:43:19.513283 2765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:19.513283 2767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:19.513741 2759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:19.514056 2759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:19.515225 2759 hybrid_clock.cc:648] HybridClock initialized: now 1790271799515215 us; error 36 us; skew 500 ppm
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:19.517752 2759 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:19.518217 2759 webserver.cc:460] Webserver started at http://127.1.8.190:39063/ using document root <none> and password file <none>
I20260924 17:43:19.518509 2759 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:19.518579 2759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:19.518743 2759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:19.519831 2759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "bcefefb3d218494e90263abdcfa67b32"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "d1faeb1b71460bc725f8cf9b8eae0ec3"
server_key_iv: "f4679c86814cdb16e766c123d8310c35"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.520206 2759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "bcefefb3d218494e90263abdcfa67b32"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "d1faeb1b71460bc725f8cf9b8eae0ec3"
server_key_iv: "f4679c86814cdb16e766c123d8310c35"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.521608 2759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:19.522384 2774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:19.522655 2759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:19.522774 2759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "bcefefb3d218494e90263abdcfa67b32"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "d1faeb1b71460bc725f8cf9b8eae0ec3"
server_key_iv: "f4679c86814cdb16e766c123d8310c35"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.522866 2759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:19.533388 2759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:19.535825 2759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:19.535979 2759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:19.540716 2759 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:34169
I20260924 17:43:19.540827 2836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:34169 every 8 connection(s)
I20260924 17:43:19.541244 2759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:19.541947 2837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:19.544452 2837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Bootstrap starting.
I20260924 17:43:19.545086 2837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:19.545388 2837 log.cc:824] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:19.546118 2837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: No bootstrap required, opened a new log
I20260924 17:43:19.547134 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 2759
I20260924 17:43:19.547223 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:19.547389 1058 external_mini_cluster.cc:1428] Setting key fbd0c1315b6c21ed0fd2e5b1a48424e9
I20260924 17:43:19.548803 2837 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcefefb3d218494e90263abdcfa67b32" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34169 } }
I20260924 17:43:19.548969 2837 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:19.549011 2837 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcefefb3d218494e90263abdcfa67b32, State: Initialized, Role: FOLLOWER
I20260924 17:43:19.549146 2837 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcefefb3d218494e90263abdcfa67b32" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34169 } }
I20260924 17:43:19.549230 2837 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:19.549284 2837 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:19.549338 2837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:19.549911 2837 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcefefb3d218494e90263abdcfa67b32" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34169 } }
I20260924 17:43:19.550029 2837 leader_election.cc:302] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcefefb3d218494e90263abdcfa67b32; no voters:
I20260924 17:43:19.550225 2837 leader_election.cc:288] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:19.550256 2842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:19.550364 2842 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [term 1 LEADER]: Becoming Leader. State: Replica: bcefefb3d218494e90263abdcfa67b32, State: Running, Role: LEADER
I20260924 17:43:19.550477 2842 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcefefb3d218494e90263abdcfa67b32" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34169 } }
I20260924 17:43:19.550521 2837 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:19.550822 2844 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcefefb3d218494e90263abdcfa67b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcefefb3d218494e90263abdcfa67b32" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34169 } } }
I20260924 17:43:19.551028 2844 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:19.550969 2843 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcefefb3d218494e90263abdcfa67b32. Latest consensus state: current_term: 1 leader_uuid: "bcefefb3d218494e90263abdcfa67b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcefefb3d218494e90263abdcfa67b32" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34169 } } }
I20260924 17:43:19.551297 2843 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:19.551486 2851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:19.552007 2851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:19.553933 2851 catalog_manager.cc:1347] Generated new cluster ID: ff23c212409f4387a1ba9f28541c4f19
I20260924 17:43:19.553993 2851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:19.560333 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.547960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.1.8.190:34169)
0924 17:43:19.548154 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:19.548163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:19.548190 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:43:19.548780 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:19.548790 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:19.548846 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:19.548995 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:19.549735 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.551009 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.552124 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.552780 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.552938 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.555879 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:19.555884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:19.555884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:19.555898 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:19.558443 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.558871 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.558874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.558874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.558999 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.559219 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.559220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.559220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.559540 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:19.559696 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:19.559861 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 17:43:19.560094 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260924 17:43:19.575831 2851 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:19.576659 2851 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:19.581693 2851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Generated new TSK 0
I20260924 17:43:19.581918 2851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:19.618311 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:34169
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:19.703517 2863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:19.703747 2863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:19.703775 2863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:19.704962 2863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:19.705004 2863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:19.705024 2863 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 17:43:19.705039 2863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:43:19.705055 2863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:19.706497 2863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:19.706689 2863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:19.706882 2863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:19.712395 2871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:19.712415 2868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:19.712420 2869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:19.712822 2863 server_base.cc:1034] running on GCE node
I20260924 17:43:19.713056 2863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:19.713269 2863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:19.714463 2863 hybrid_clock.cc:648] HybridClock initialized: now 1790271799714455 us; error 35 us; skew 500 ppm
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:19.717067 2863 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:19.717572 2863 webserver.cc:460] Webserver started at http://127.1.8.129:34511/ using document root <none> and password file <none>
I20260924 17:43:19.717813 2863 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:19.717881 2863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:19.718037 2863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:19.719265 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "3614654fc5184f2e8dc47da6d498926b"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "f619fafd954698ab7443c02c74fad41c"
server_key_iv: "7e3049ad6dfd049a74cfe1381c34d58d"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.719638 2863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "3614654fc5184f2e8dc47da6d498926b"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "f619fafd954698ab7443c02c74fad41c"
server_key_iv: "7e3049ad6dfd049a74cfe1381c34d58d"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.721009 2863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:43:19.721863 2878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:19.722060 2863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:19.722167 2863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "3614654fc5184f2e8dc47da6d498926b"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "f619fafd954698ab7443c02c74fad41c"
server_key_iv: "7e3049ad6dfd049a74cfe1381c34d58d"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.722244 2863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:19.731432 2863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:19.733374 2863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:19.733537 2863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:19.734126 2863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:19.734185 2863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:19.734220 2863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:19.734247 2863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:19.741673 2863 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:44805
I20260924 17:43:19.741758 2993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:44805 every 8 connection(s)
I20260924 17:43:19.742118 2863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:19.743475 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 2863
I20260924 17:43:19.743558 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:19.743781 1058 external_mini_cluster.cc:1428] Setting key dc33d0d7bf6cb2815e69ea065ed0fe36
I20260924 17:43:19.746549 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.734528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56567 (local address 127.1.8.190:34169)
0924 17:43:19.734603 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:19.734606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:19.734878 (+ 272us) server_negotiation.cc:407] Connection header received
0924 17:43:19.735643 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:19.735644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:19.735662 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:19.735726 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:19.736684 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.737271 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.738925 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.739158 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.739244 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.742859 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:19.742860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:19.742861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:19.742868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:19.744987 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.745485 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.745486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.745486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.745552 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.745828 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.745829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.745830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.745955 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:19.746026 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:19.746337 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 17:43:19.746458 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:43:19.746780 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.734542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.1.8.129:56567)
0924 17:43:19.734708 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:19.734762 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:19.735500 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:19.735816 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:19.735821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:19.736247 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:19.736563 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.736570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.737340 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.737341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.738834 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.738838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.739240 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.739241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.739417 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.740630 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:19.740643 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:19.742756 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:19.745092 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.745102 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.745105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.745370 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.745627 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.745627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.745628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.745756 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.746027 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:19.746034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:19.746192 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 17:43:19.746295 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 17:43:19.746494 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260924 17:43:19.747288 2994 heartbeater.cc:347] Connected to a master server at 127.1.8.190:34169
I20260924 17:43:19.747411 2994 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:19.747603 2994 heartbeater.cc:511] Master 127.1.8.190:34169 requested a full tablet report, sending...
I20260924 17:43:19.748179 2790 ts_manager.cc:194] Registered new tserver with Master: 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805)
I20260924 17:43:19.749126 2790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:56567
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:43:19.755662 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.748709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:41663 (local address 127.1.8.190:34169)
0924 17:43:19.748744 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:19.748746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:19.748787 (+ 41us) server_negotiation.cc:407] Connection header received
0924 17:43:19.748877 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:19.748878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:19.748886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:19.748940 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:19.749384 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.750307 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.751574 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.751760 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.751819 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.752982 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:19.752983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:19.752983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:19.752989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:19.754551 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.754894 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.754895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.754895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.754947 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.755207 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.755208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.755208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.755313 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:19.755384 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:19.755536 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 17:43:19.755600 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 17:43:19.755710 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.748553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.1.8.129:41663)
0924 17:43:19.748712 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:19.748727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:19.748809 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:19.748946 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:19.748947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:19.749180 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:19.749268 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.749271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.750478 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.750479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.751490 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.751493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.751850 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.751851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.751938 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.752383 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:19.752389 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:19.752882 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:19.754635 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.754635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.754636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.754823 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.754989 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.754989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.754989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.755034 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.755394 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:19.755395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:19.755447 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 17:43:19.755506 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:43:19.755596 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:19.773459 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:34169
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:19.856344 3003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:19.856577 3003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:19.856606 3003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:19.857823 3003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:19.857872 3003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:19.857893 3003 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 17:43:19.857905 3003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:43:19.857924 3003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:19.859486 3003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:19.859767 3003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:19.860013 3003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:19.865208 3009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:19.865222 3008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:19.865340 3003 server_base.cc:1034] running on GCE node
W20260924 17:43:19.865222 3011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:19.865646 3003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:19.865922 3003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:19.867086 3003 hybrid_clock.cc:648] HybridClock initialized: now 1790271799867069 us; error 47 us; skew 500 ppm
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:19.869593 3003 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:19.870028 3003 webserver.cc:460] Webserver started at http://127.1.8.130:44029/ using document root <none> and password file <none>
I20260924 17:43:19.870220 3003 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:19.870261 3003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:19.870354 3003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:19.871465 3003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "41325bcff795d14f549477be1dba6c44"
server_key_iv: "278835f2549680c7e94359bd1aa7171b"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.871775 3003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "41325bcff795d14f549477be1dba6c44"
server_key_iv: "278835f2549680c7e94359bd1aa7171b"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.873039 3003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:19.873690 3018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:19.873865 3003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:19.873945 3003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d"
format_stamp: "Formatted at 2026-09-24 17:43:19 on dist-test-slave-r7hf"
server_key: "41325bcff795d14f549477be1dba6c44"
server_key_iv: "278835f2549680c7e94359bd1aa7171b"
server_key_version: "encryptionkey@0"
I20260924 17:43:19.874003 3003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:19.904848 3003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:19.907334 3003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:19.907480 3003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:19.907958 3003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:19.908001 3003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:19.908027 3003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:19.908039 3003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:43:19 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:19.915308 3003 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:36753
I20260924 17:43:19.915396 3133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:36753 every 8 connection(s)
I20260924 17:43:19.915737 3003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:19.918262 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.908503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:37237 (local address 127.1.8.190:34169)
0924 17:43:19.908583 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:19.908586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:19.908816 (+ 230us) server_negotiation.cc:407] Connection header received
0924 17:43:19.909314 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:19.909315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:19.909331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:19.909403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:19.910299 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.911174 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.912245 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.912444 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.912531 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.914798 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:19.914799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:19.914799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:19.914808 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:19.916679 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.917175 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.917177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.917177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.917246 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.917531 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.917532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.917532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.917674 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:19.917770 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:19.918069 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 17:43:19.918159 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:43:19.918500 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.908489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.1.8.130:37237)
0924 17:43:19.908677 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:19.908720 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:19.909213 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:19.909517 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:19.909521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:19.909886 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:19.910188 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.910193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.911276 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.911277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.912126 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.912128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.912556 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.912557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.912695 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.913252 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:19.913261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:19.914684 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:19.916794 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.916801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.916803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.917052 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.917320 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.917321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.917321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.917442 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.917759 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:19.917765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:19.917920 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 17:43:19.918045 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 17:43:19.918232 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
I20260924 17:43:19.918989 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3003
I20260924 17:43:19.919050 3134 heartbeater.cc:347] Connected to a master server at 127.1.8.190:34169
I20260924 17:43:19.919111 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:19.919129 3134 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:19.919323 1058 external_mini_cluster.cc:1428] Setting key 6b1871e5ddbffb657ebe5d943790466e
I20260924 17:43:19.919334 3134 heartbeater.cc:511] Master 127.1.8.190:34169 requested a full tablet report, sending...
I20260924 17:43:19.919708 2790 ts_manager.cc:194] Registered new tserver with Master: 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753)
I20260924 17:43:19.920231 2790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:37237
I20260924 17:43:19.926591 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.920387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33301 (local address 127.1.8.190:34169)
0924 17:43:19.920429 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:19.920431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:19.920474 (+ 43us) server_negotiation.cc:407] Connection header received
0924 17:43:19.920559 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:19.920560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:19.920572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:19.920652 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:19.921024 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.921715 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.922487 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.922692 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.922786 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.923774 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:19.923776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:19.923776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:19.923783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:19.925295 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.925670 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.925671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.925671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.925731 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:19.925927 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:19.925927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:19.925928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:19.926130 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:19.926212 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:19.926418 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 17:43:19.926489 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:19.926776 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:19.920196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.1.8.130:33301)
0924 17:43:19.920405 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:19.920417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:19.920500 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:19.920667 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:19.920668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:19.920860 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:19.920927 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.920929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.921824 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.921825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.922338 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:19.922340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:19.922787 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:19.922788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:19.922872 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:19.923346 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:19.923351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:19.923671 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:19.925369 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.925371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.925371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.925583 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.925798 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:19.925799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:19.925800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:19.925857 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:19.926238 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:19.926239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:19.926280 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:43:19.926358 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:43:19.926490 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:19.946165 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:34169
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:20.030926 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:20.031149 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:20.031177 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:20.032351 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:20.032392 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:20.032414 3143 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 17:43:20.032426 3143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:43:20.032444 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:20.033854 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:34169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:20.034055 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:20.034240 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:20.039513 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:20.039508 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:20.039513 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:20.040030 3143 server_base.cc:1034] running on GCE node
I20260924 17:43:20.040199 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:20.040412 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:20.041579 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790271800041567 us; error 27 us; skew 500 ppm
Sep 24 17:43:20 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271800, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:20.044281 3143 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:20.044788 3143 webserver.cc:460] Webserver started at http://127.1.8.131:45783/ using document root <none> and password file <none>
I20260924 17:43:20.045024 3143 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:20.045094 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:20.045250 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:20.046365 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "2ab2435a50bb4f9b93d717668f15a07b"
format_stamp: "Formatted at 2026-09-24 17:43:20 on dist-test-slave-r7hf"
server_key: "a9d35fed441270fd8d69b1b5d097035c"
server_key_iv: "cf68a4e50166c90c16ca2ede9d37c788"
server_key_version: "encryptionkey@0"
I20260924 17:43:20.046936 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab2435a50bb4f9b93d717668f15a07b"
format_stamp: "Formatted at 2026-09-24 17:43:20 on dist-test-slave-r7hf"
server_key: "a9d35fed441270fd8d69b1b5d097035c"
server_key_iv: "cf68a4e50166c90c16ca2ede9d37c788"
server_key_version: "encryptionkey@0"
I20260924 17:43:20.048435 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:20.049226 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:20.049399 3143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:20.049507 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "2ab2435a50bb4f9b93d717668f15a07b"
format_stamp: "Formatted at 2026-09-24 17:43:20 on dist-test-slave-r7hf"
server_key: "a9d35fed441270fd8d69b1b5d097035c"
server_key_iv: "cf68a4e50166c90c16ca2ede9d37c788"
server_key_version: "encryptionkey@0"
I20260924 17:43:20.049584 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:20.064275 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:20.066792 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:20.066965 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:20.067517 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:20.067580 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:20.067626 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:20.067651 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:43:20 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271800, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:20.075160 3143 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:37963
I20260924 17:43:20.075295 3273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:37963 every 8 connection(s)
I20260924 17:43:20.075629 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:43:20.078608 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.068136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:54771 (local address 127.1.8.190:34169)
0924 17:43:20.068216 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.068218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.068587 (+ 369us) server_negotiation.cc:407] Connection header received
0924 17:43:20.069289 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.069290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.069305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.069385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:20.070364 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.070900 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.072610 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.072968 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.073050 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.075496 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:20.075498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:20.075498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:20.075507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:20.077061 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.077497 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.077499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.077499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.077559 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.077879 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.077881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.077881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.078026 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:20.078091 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.078467 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.078541 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:43:20.078846 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.068114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.1.8.131:54771)
0924 17:43:20.068328 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.068379 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.069150 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.069486 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.069491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.069910 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:20.070232 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.070275 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.070982 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.070983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.072492 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.072495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.073060 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.073061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.073212 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.073819 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:20.073829 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:20.075389 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:20.077134 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:20.077140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:20.077141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:20.077401 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:20.077628 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:20.077628 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:20.077629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:20.077727 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:20.078124 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:20.078129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:20.078255 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.078371 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.078584 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260924 17:43:20.079416 3274 heartbeater.cc:347] Connected to a master server at 127.1.8.190:34169
I20260924 17:43:20.079493 3274 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:20.079684 3274 heartbeater.cc:511] Master 127.1.8.190:34169 requested a full tablet report, sending...
I20260924 17:43:20.080142 2791 ts_manager.cc:194] Registered new tserver with Master: 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:20.080698 2791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:54771
I20260924 17:43:20.081593 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3143
I20260924 17:43:20.081679 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:20.081898 1058 external_mini_cluster.cc:1428] Setting key 83f975c76e385ad7a7439b9ffabd2976
I20260924 17:43:20.082727 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:43:20.087113 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.080520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.1.8.131:59745)
0924 17:43:20.080705 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.080715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.080845 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.081001 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.081002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.081178 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:20.081234 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.081235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.082048 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.082049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.082770 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.082773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.083146 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.083147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.083227 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.083739 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:20.083744 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:20.084066 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:20.086030 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:20.086031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:20.086032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:20.086204 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:20.086446 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:20.086447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:20.086447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:20.086510 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:20.086861 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:20.086862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:20.086900 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.086951 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.087035 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260924 17:43:20.087258 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.080593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:59745 (local address 127.1.8.190:34169)
0924 17:43:20.080644 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.080646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.080798 (+ 152us) server_negotiation.cc:407] Connection header received
0924 17:43:20.080910 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.080911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.080922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.080975 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:20.081309 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.081969 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.082863 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.083065 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.083146 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.084196 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:20.084197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:20.084197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:20.084204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:20.085942 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.086284 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.086286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.086286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.086349 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.086574 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.086576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.086576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.086790 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:20.086874 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.087032 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.087127 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 17:43:20 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:43:20 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:20.107133 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.100053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.1.8.190:34169)
0924 17:43:20.100130 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.100132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.100142 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:20.100174 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.100175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.100185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.100241 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:20.100822 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.101305 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.102024 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.102210 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.102272 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.104585 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:20.104587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:20.104587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:20.104594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:20.106069 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.106435 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.106437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.106437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.106507 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.106688 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.106689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.106689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.106822 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:20.106903 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.106981 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.107039 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 17:43:20.108848 2791 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:20.110467 2791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:20.116777 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.112842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44805 (local address 127.0.0.1:42062)
0924 17:43:20.112993 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.113057 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.113130 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.113350 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.113353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.113355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.113438 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.113440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.114568 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.114569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.115553 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.115555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.116472 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.116473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.116552 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.116566 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.116642 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.116691 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":55,"thread_start_us":63,"threads_started":1}
I20260924 17:43:20.116802 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.112939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.1.8.129:44805)
0924 17:43:20.113120 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.113125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.113138 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:43:20.113223 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.113225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.113240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.113345 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.113651 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.114478 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.115642 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.116380 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.116456 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.116515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.116664 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.116703 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260924 17:43:20.117488 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.112851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:37963 (local address 127.0.0.1:39648)
0924 17:43:20.113154 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.113166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.113218 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.113531 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.113532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.113534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.113570 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.113571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.114593 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.114594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.116086 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.116088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.117244 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.117244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.117302 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.117306 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.117378 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.117417 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":29,"thread_start_us":90,"threads_started":1}
I20260924 17:43:20.117522 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.113185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.1.8.131:37963)
0924 17:43:20.113352 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.113357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.113368 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:20.113407 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.113409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.113425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.113511 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.113641 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.114511 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.116175 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.116855 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.117145 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.117214 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.117392 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.117429 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260924 17:43:20.117575 2928 tablet_service.cc:1467] Processing CreateTablet for tablet 84b90b5b0a394c5b8581d8e19f51c028 (DEFAULT_TABLE table=test-table [id=4955db6ee6ca446c9b3d5067036abef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:20.117985 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b90b5b0a394c5b8581d8e19f51c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.118191 3208 tablet_service.cc:1467] Processing CreateTablet for tablet 84b90b5b0a394c5b8581d8e19f51c028 (DEFAULT_TABLE table=test-table [id=4955db6ee6ca446c9b3d5067036abef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:20.118594 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.113048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36753 (local address 127.0.0.1:37954)
0924 17:43:20.113598 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.113607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.113650 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.114793 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.114794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.114797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.114835 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.114838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.115994 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.115995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.117188 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.117191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.118341 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.118341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.118428 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.118433 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.118481 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.118519 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":66,"threads_started":1}
I20260924 17:43:20.118582 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b90b5b0a394c5b8581d8e19f51c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.118816 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.113443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.1.8.130:36753)
0924 17:43:20.114623 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.114628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.114639 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:20.114677 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.114679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.114695 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.114784 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.114895 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.115918 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.117279 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.118252 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.118330 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.118491 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.118520 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.118554 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260924 17:43:20.121430 3298 tablet_bootstrap.cc:492] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Bootstrap starting.
I20260924 17:43:20.121730 3068 tablet_service.cc:1467] Processing CreateTablet for tablet 84b90b5b0a394c5b8581d8e19f51c028 (DEFAULT_TABLE table=test-table [id=4955db6ee6ca446c9b3d5067036abef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:20.122027 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b90b5b0a394c5b8581d8e19f51c028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.122051 3298 tablet_bootstrap.cc:654] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.122337 3298 log.cc:824] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:20.121385 3297 tablet_bootstrap.cc:492] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Bootstrap starting.
I20260924 17:43:20.123202 3298 tablet_bootstrap.cc:492] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: No bootstrap required, opened a new log
I20260924 17:43:20.123273 3298 ts_tablet_manager.cc:1397] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:43:20.123392 3297 tablet_bootstrap.cc:654] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.123729 3297 log.cc:824] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:20.124603 3297 tablet_bootstrap.cc:492] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: No bootstrap required, opened a new log
I20260924 17:43:20.124671 3297 ts_tablet_manager.cc:1397] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:20.125429 3301 tablet_bootstrap.cc:492] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Bootstrap starting.
I20260924 17:43:20.125476 3298 raft_consensus.cc:348] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.125656 3298 raft_consensus.cc:374] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.125698 3298 raft_consensus.cc:729] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3614654fc5184f2e8dc47da6d498926b, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.125842 3298 consensus_queue.cc:260] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.126107 3298 ts_tablet_manager.cc:1428] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:20.126147 3301 tablet_bootstrap.cc:654] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.126232 2994 heartbeater.cc:503] Master 127.1.8.190:34169 was elected leader, sending a full tablet report...
I20260924 17:43:20.126549 3301 log.cc:824] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:20.126957 3297 raft_consensus.cc:348] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.127112 3297 raft_consensus.cc:374] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.127156 3297 raft_consensus.cc:729] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.127300 3301 tablet_bootstrap.cc:492] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: No bootstrap required, opened a new log
I20260924 17:43:20.127267 3297 consensus_queue.cc:260] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.127365 3301 ts_tablet_manager.cc:1397] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:43:20.127532 3297 ts_tablet_manager.cc:1428] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:20.127549 3274 heartbeater.cc:503] Master 127.1.8.190:34169 was elected leader, sending a full tablet report...
I20260924 17:43:20.129200 3301 raft_consensus.cc:348] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.129326 3301 raft_consensus.cc:374] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.129347 3301 raft_consensus.cc:729] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.129475 3301 consensus_queue.cc:260] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.129733 3301 ts_tablet_manager.cc:1428] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:43:20.129776 3134 heartbeater.cc:503] Master 127.1.8.190:34169 was elected leader, sending a full tablet report...
W20260924 17:43:20.166378 3135 tablet.cc:2369] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:43:20.243080 2995 tablet.cc:2369] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:43:20.276122 3304 raft_consensus.cc:482] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:20.276320 3304 raft_consensus.cc:504] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.276796 3304 leader_election.cc:288] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753)
I20260924 17:43:20.280723 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.277311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:44825 (local address 127.1.8.130:36753)
0924 17:43:20.277357 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.277359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.277486 (+ 127us) server_negotiation.cc:407] Connection header received
0924 17:43:20.277552 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.277553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.277567 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.277618 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.277780 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.278485 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.279926 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.280310 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.280370 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.280380 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.280608 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.280652 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:20.280743 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.277210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36753 (local address 127.1.8.131:44825)
0924 17:43:20.277426 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.277437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.277491 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.277639 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.277639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.277641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.277694 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.277695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.278565 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.278566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.279829 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.279832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.280404 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.280405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.280494 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.280499 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.280599 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.280643 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":11,"thread_start_us":125,"threads_started":1}
I20260924 17:43:20.280838 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.277074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44805 (local address 127.1.8.131:36161)
0924 17:43:20.277156 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.277171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.277250 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.277449 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.277450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.277452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.277525 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.277527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.278565 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.278566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.279848 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.279851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.280476 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.280477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.280602 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.280605 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.280684 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.280729 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":106}
I20260924 17:43:20.281168 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.277107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:36161 (local address 127.1.8.129:44805)
0924 17:43:20.277183 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.277185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.277229 (+ 44us) server_negotiation.cc:407] Connection header received
0924 17:43:20.277329 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.277330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.277345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.277410 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.277611 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.278436 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.279983 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.280389 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.280465 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.280476 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.280694 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.280805 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:43:20.281388 3088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b90b5b0a394c5b8581d8e19f51c028" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" is_pre_election: true
I20260924 17:43:20.281469 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b90b5b0a394c5b8581d8e19f51c028" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b" is_pre_election: true
I20260924 17:43:20.281577 3088 raft_consensus.cc:2454] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 0.
I20260924 17:43:20.281635 2948 raft_consensus.cc:2454] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 0.
I20260924 17:43:20.281900 3160 leader_election.cc:302] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab2435a50bb4f9b93d717668f15a07b, 3614654fc5184f2e8dc47da6d498926b; no voters:
I20260924 17:43:20.282063 3304 raft_consensus.cc:2781] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:20.282117 3304 raft_consensus.cc:482] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:20.282146 3304 raft_consensus.cc:3037] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.283298 3304 raft_consensus.cc:504] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.283488 3304 leader_election.cc:288] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 election: Requested vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753)
I20260924 17:43:20.283773 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b90b5b0a394c5b8581d8e19f51c028" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b"
I20260924 17:43:20.283885 2948 raft_consensus.cc:3037] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.284996 2948 raft_consensus.cc:2454] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 1.
I20260924 17:43:20.285225 3160 leader_election.cc:302] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab2435a50bb4f9b93d717668f15a07b, 3614654fc5184f2e8dc47da6d498926b; no voters:
I20260924 17:43:20.285389 3304 raft_consensus.cc:2781] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:20.285614 3304 raft_consensus.cc:686] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 LEADER]: Becoming Leader. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Running, Role: LEADER
I20260924 17:43:20.285754 3304 consensus_queue.cc:237] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } }
I20260924 17:43:20.286362 3088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b90b5b0a394c5b8581d8e19f51c028" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d"
I20260924 17:43:20.286487 3088 raft_consensus.cc:3037] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.287393 3088 raft_consensus.cc:2454] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 1.
I20260924 17:43:20.287499 2791 catalog_manager.cc:5495] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131). New cstate: current_term: 1 leader_uuid: "2ab2435a50bb4f9b93d717668f15a07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } health_report { overall_health: UNKNOWN } } }
Sep 24 17:43:20 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271799, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
W20260924 17:43:20.326576 3275 tablet.cc:2369] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:43:20.327068 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.319399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34169 (local address 127.0.0.1:46688)
0924 17:43:20.319575 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.319588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.319689 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.319938 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.319939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.320265 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:20.320316 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.320318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.320975 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.320976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.321549 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.321551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.321959 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.321960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.322000 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.322559 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:20.322564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:20.323995 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:20.325772 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:20.325781 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:20.325783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:20.325998 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:20.326296 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:20.326297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:20.326297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:20.326375 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:20.326651 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:20.326660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:20.326756 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.326835 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.326968 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260924 17:43:20.327136 2841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.319474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.1.8.190:34169)
0924 17:43:20.319558 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.319561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.319689 (+ 128us) server_negotiation.cc:407] Connection header received
0924 17:43:20.319841 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.319842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.319851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.319915 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:20.320396 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.320888 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.321623 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.321884 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.321948 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.324089 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:20.324091 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:20.324091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:20.324097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:20.325684 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.326098 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.326099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.326104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.326199 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:20.326455 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:20.326456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:20.326456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:20.326582 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:20.326635 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.326874 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.327016 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":19}
I20260924 17:43:20.329535 2791 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:46688:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 17:43:20.329815 2791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:20.331483 3068 tablet_service.cc:1467] Processing CreateTablet for tablet cf9b7bc7b0ea437b8b2c719bd6af4030 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5514268fce514d89a8e2bce52027f3e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:43:20.331460 2928 tablet_service.cc:1467] Processing CreateTablet for tablet cf9b7bc7b0ea437b8b2c719bd6af4030 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5514268fce514d89a8e2bce52027f3e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:43:20.331646 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9b7bc7b0ea437b8b2c719bd6af4030. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.331646 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9b7bc7b0ea437b8b2c719bd6af4030. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.333235 3301 tablet_bootstrap.cc:492] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Bootstrap starting.
I20260924 17:43:20.333403 3208 tablet_service.cc:1467] Processing CreateTablet for tablet cf9b7bc7b0ea437b8b2c719bd6af4030 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5514268fce514d89a8e2bce52027f3e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:43:20.333541 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf9b7bc7b0ea437b8b2c719bd6af4030. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.333931 3301 tablet_bootstrap.cc:654] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.334257 3298 tablet_bootstrap.cc:492] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: Bootstrap starting.
I20260924 17:43:20.334640 3298 tablet_bootstrap.cc:654] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.334640 3301 tablet_bootstrap.cc:492] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: No bootstrap required, opened a new log
I20260924 17:43:20.334684 3301 ts_tablet_manager.cc:1397] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:20.334825 3297 tablet_bootstrap.cc:492] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: Bootstrap starting.
I20260924 17:43:20.334882 3301 raft_consensus.cc:348] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.334968 3301 raft_consensus.cc:374] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.334992 3301 raft_consensus.cc:729] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.335041 3301 consensus_queue.cc:260] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.335130 3301 ts_tablet_manager.cc:1428] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:20.335250 3297 tablet_bootstrap.cc:654] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.335338 3298 tablet_bootstrap.cc:492] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: No bootstrap required, opened a new log
I20260924 17:43:20.335304 3305 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.335379 3298 ts_tablet_manager.cc:1397] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:20.335393 3305 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.335536 3298 raft_consensus.cc:348] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.335613 3298 raft_consensus.cc:374] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.335635 3298 raft_consensus.cc:729] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3614654fc5184f2e8dc47da6d498926b, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.335680 3298 consensus_queue.cc:260] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.335794 3298 ts_tablet_manager.cc:1428] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:20.335860 3297 tablet_bootstrap.cc:492] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: No bootstrap required, opened a new log
I20260924 17:43:20.335896 3297 ts_tablet_manager.cc:1397] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:20.335850 3302 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.335943 3302 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.336035 3297 raft_consensus.cc:348] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.336118 3297 raft_consensus.cc:374] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.336148 3297 raft_consensus.cc:729] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.336196 3297 consensus_queue.cc:260] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.336282 3297 ts_tablet_manager.cc:1428] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:20.336490 3304 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.336581 3304 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.539146 3305 raft_consensus.cc:482] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:20.539299 3305 raft_consensus.cc:504] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.539721 3305 leader_election.cc:288] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:20.543464 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.539981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:36195 (local address 127.1.8.129:44805)
0924 17:43:20.540033 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.540035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.540188 (+ 153us) server_negotiation.cc:407] Connection header received
0924 17:43:20.540224 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.540225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.540234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.540304 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.540505 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.541046 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.542643 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.543092 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.543273 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.543287 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.543353 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.543393 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:20.543478 3319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.539910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44805 (local address 127.1.8.130:36195)
0924 17:43:20.540049 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.540060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.540148 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.540327 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.540328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.540330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.540417 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.540419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.541140 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.541141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.542548 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.542552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.543200 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.543201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.543290 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.543295 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.543350 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.543397 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":67,"thread_start_us":66,"threads_started":1}
I20260924 17:43:20.543532 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.539983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:41033 (local address 127.1.8.131:37963)
0924 17:43:20.540048 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.540050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.540128 (+ 78us) server_negotiation.cc:407] Connection header received
0924 17:43:20.540198 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.540199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.540216 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.540259 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:20.540420 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.541101 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.542682 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.543053 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.543158 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.543172 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.543422 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.543465 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:43:20.543785 3318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.539900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:37963 (local address 127.1.8.130:41033)
0924 17:43:20.540040 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.540051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.540119 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.540283 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.540284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.540287 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:20.540346 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.540348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.541186 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.541187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.542548 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.542552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.543238 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.543240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.543317 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.543323 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.543402 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.543448 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":76,"thread_start_us":81,"threads_started":1}
I20260924 17:43:20.543854 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9b7bc7b0ea437b8b2c719bd6af4030" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b" is_pre_election: true
I20260924 17:43:20.543963 2948 raft_consensus.cc:2454] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 0.
I20260924 17:43:20.544135 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9b7bc7b0ea437b8b2c719bd6af4030" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab2435a50bb4f9b93d717668f15a07b" is_pre_election: true
I20260924 17:43:20.544168 3020 leader_election.cc:302] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3614654fc5184f2e8dc47da6d498926b, 6cde7eaf7ba44f46999a9c6edfdcc47d; no voters:
I20260924 17:43:20.544268 3228 raft_consensus.cc:2454] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 0.
I20260924 17:43:20.544324 3305 raft_consensus.cc:2781] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:20.544379 3305 raft_consensus.cc:482] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:20.544400 3305 raft_consensus.cc:3037] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.545087 3305 raft_consensus.cc:504] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.545209 3305 leader_election.cc:288] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 election: Requested vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:20.545387 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9b7bc7b0ea437b8b2c719bd6af4030" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b"
I20260924 17:43:20.545469 2948 raft_consensus.cc:3037] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.546103 2948 raft_consensus.cc:2454] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 1.
I20260924 17:43:20.546232 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf9b7bc7b0ea437b8b2c719bd6af4030" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab2435a50bb4f9b93d717668f15a07b"
I20260924 17:43:20.546271 3020 leader_election.cc:302] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3614654fc5184f2e8dc47da6d498926b, 6cde7eaf7ba44f46999a9c6edfdcc47d; no voters:
I20260924 17:43:20.546335 3228 raft_consensus.cc:3037] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.546383 3305 raft_consensus.cc:2781] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:20.546535 3305 raft_consensus.cc:686] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 LEADER]: Becoming Leader. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Running, Role: LEADER
I20260924 17:43:20.546627 3305 consensus_queue.cc:237] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.547132 3320 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: TxnStatusTablet state changed. Reason: New leader 6cde7eaf7ba44f46999a9c6edfdcc47d. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.547217 3320 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:43:20.547277 3228 raft_consensus.cc:2454] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 1.
I20260924 17:43:20.547518 2791 catalog_manager.cc:5495] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130). New cstate: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260924 17:43:20.547873 3322 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 17:43:20.548043 3322 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 17:43:20.566843 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.563855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.1.8.130:36753)
0924 17:43:20.563923 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.563925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.563942 (+ 17us) server_negotiation.cc:407] Connection header received
0924 17:43:20.563979 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.563980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.563998 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.564069 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:20.564254 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.564756 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.565897 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.566127 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.566194 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.566494 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.566608 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.566688 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.566755 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.566778 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:43:20.566901 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.563670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36753 (local address 127.0.0.1:37958)
0924 17:43:20.563755 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.563766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.563819 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.564098 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.564099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.564101 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:20.564157 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.564159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.564840 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.564841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.565766 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.565769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.566211 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.566212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.566283 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.566304 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.566685 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.566693 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.566752 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.566818 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 17:43:20.568979 3228 raft_consensus.cc:1261] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Refusing update from remote peer 6cde7eaf7ba44f46999a9c6edfdcc47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:20.569135 2948 raft_consensus.cc:1261] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Refusing update from remote peer 6cde7eaf7ba44f46999a9c6edfdcc47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:20.569312 3305 consensus_queue.cc:1035] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:20.569561 3305 consensus_queue.cc:1035] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:20.570691 3304 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: TxnStatusTablet state changed. Reason: New leader 6cde7eaf7ba44f46999a9c6edfdcc47d. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.570798 3304 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.571636 3302 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: TxnStatusTablet state changed. Reason: New leader 6cde7eaf7ba44f46999a9c6edfdcc47d. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.571736 3302 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.572386 3320 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.572486 3320 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:43:20.572490 3326 mvcc.cc:205] Tried to move back new op lower bound from 7332953295127498752 to 7332953295039320064. Current Snapshot: MvccSnapshot[applied={T|T < 7332953295127498752}]
I20260924 17:43:20.572811 3302 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.572899 3302 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 3614654fc5184f2e8dc47da6d498926b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.575111 3305 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.575198 3305 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 6cde7eaf7ba44f46999a9c6edfdcc47d: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:43:20.575567 3304 tablet_replica.cc:439] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } } }
I20260924 17:43:20.575654 3304 tablet_replica.cc:442] T cf9b7bc7b0ea437b8b2c719bd6af4030 P 2ab2435a50bb4f9b93d717668f15a07b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:43:20.575886 3327 mvcc.cc:205] Tried to move back new op lower bound from 7332953295127498752 to 7332953295039320064. Current Snapshot: MvccSnapshot[applied={T|T < 7332953295127498752}]
I20260924 17:43:20.577414 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.574257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.1.8.131:37963)
0924 17:43:20.574308 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.574310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.574343 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:43:20.574470 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.574471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.574492 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.574577 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:20.574736 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.575460 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.576469 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.576778 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.576859 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.577051 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.577174 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.577273 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.577320 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.577344 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:43:20.582446 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.579945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36753 (local address 127.1.8.131:38265)
0924 17:43:20.580043 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.580055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.580105 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.580261 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.580262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.580263 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:20.580326 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.580327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.581019 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.581020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.581609 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.581611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.581979 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.581980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.582050 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.582063 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.582265 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.582268 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.582309 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.582356 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 17:43:20.582446 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.580050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:38265 (local address 127.1.8.130:36753)
0924 17:43:20.580089 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.580091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.580100 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:20.580168 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.580170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.580180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.580249 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:20.580380 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.580937 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.581701 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.581895 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.581980 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.582158 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.582199 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.582264 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.582321 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.582341 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:20.585191 2948 raft_consensus.cc:1261] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Refusing update from remote peer 2ab2435a50bb4f9b93d717668f15a07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:20.585274 3088 raft_consensus.cc:1261] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Refusing update from remote peer 2ab2435a50bb4f9b93d717668f15a07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:20.585573 3304 consensus_queue.cc:1035] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:20.585742 3307 consensus_queue.cc:1035] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:20.587534 3325 mvcc.cc:205] Tried to move back new op lower bound from 7332953295195144192 to 7332953293971484672. Current Snapshot: MvccSnapshot[applied={T|T < 7332953295195144192}]
I20260924 17:43:20.596777 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.593444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:35979 (local address 127.1.8.131:37963)
0924 17:43:20.593485 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.593487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.593613 (+ 126us) server_negotiation.cc:407] Connection header received
0924 17:43:20.593706 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.593707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.593721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.593790 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:20.593977 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.594862 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.595885 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.596215 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.596276 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.596519 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.596554 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.596604 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.596666 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.596687 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:43:20.596810 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.593348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:37963 (local address 127.1.8.130:35979)
0924 17:43:20.593536 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.593546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.593607 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.593814 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.593815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.593817 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:20.593885 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.593887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.594942 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.594943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.595798 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.595800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.596313 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.596315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.596382 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.596397 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.596609 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.596611 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.596661 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.596715 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260924 17:43:20.607519 2790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:20.607659 2790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:20.608564 2790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Sending DeleteTablet for 3 replicas of tablet 84b90b5b0a394c5b8581d8e19f51c028
I20260924 17:43:20.608987 2928 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b90b5b0a394c5b8581d8e19f51c028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:20 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42062
I20260924 17:43:20.609036 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b90b5b0a394c5b8581d8e19f51c028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:20 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39648
I20260924 17:43:20.609026 3068 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b90b5b0a394c5b8581d8e19f51c028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:20 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37954
I20260924 17:43:20.609349 3346 tablet_replica.cc:330] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: stopping tablet replica
I20260924 17:43:20.609473 3346 raft_consensus.cc:2229] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:20.609543 3346 raft_consensus.cc:2258] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:20.609552 2790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:20.609717 2790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:20.610294 3346 ts_tablet_manager.cc:1905] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:20.611069 3208 tablet_service.cc:1467] Processing CreateTablet for tablet 77a1fec3a09a4545b2b6af6b297f1fd5 (DEFAULT_TABLE table=test-table [id=708dc891536b4cb98c0eeb89a80dcf05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:20.611123 3068 tablet_service.cc:1467] Processing CreateTablet for tablet 77a1fec3a09a4545b2b6af6b297f1fd5 (DEFAULT_TABLE table=test-table [id=708dc891536b4cb98c0eeb89a80dcf05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:20.611200 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a1fec3a09a4545b2b6af6b297f1fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.611245 3068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a1fec3a09a4545b2b6af6b297f1fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.612550 3297 tablet_bootstrap.cc:492] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Bootstrap starting.
I20260924 17:43:20.612553 3301 tablet_bootstrap.cc:492] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Bootstrap starting.
I20260924 17:43:20.612959 3297 tablet_bootstrap.cc:654] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.612979 3301 tablet_bootstrap.cc:654] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.613505 2928 tablet_service.cc:1467] Processing CreateTablet for tablet 77a1fec3a09a4545b2b6af6b297f1fd5 (DEFAULT_TABLE table=test-table [id=708dc891536b4cb98c0eeb89a80dcf05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:20.613633 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a1fec3a09a4545b2b6af6b297f1fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:20.614748 3345 tablet_replica.cc:330] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: stopping tablet replica
I20260924 17:43:20.614892 3345 raft_consensus.cc:2229] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:20.614964 3345 raft_consensus.cc:2258] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:20.614998 3298 tablet_bootstrap.cc:492] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Bootstrap starting.
I20260924 17:43:20.615121 3344 tablet_replica.cc:330] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: stopping tablet replica
I20260924 17:43:20.615417 3298 tablet_bootstrap.cc:654] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:20.615515 3344 raft_consensus.cc:2229] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:20.615648 3344 raft_consensus.cc:2258] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:20.616112 3344 ts_tablet_manager.cc:1905] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:20.616550 3345 ts_tablet_manager.cc:1905] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:20.617604 3346 ts_tablet_manager.cc:1918] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:20.617691 3346 log.cc:1196] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/84b90b5b0a394c5b8581d8e19f51c028
I20260924 17:43:20.617977 3346 ts_tablet_manager.cc:1939] T 84b90b5b0a394c5b8581d8e19f51c028 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting consensus metadata
I20260924 17:43:20.617983 3344 ts_tablet_manager.cc:1918] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:20.618057 3344 log.cc:1196] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/84b90b5b0a394c5b8581d8e19f51c028
I20260924 17:43:20.618307 3344 ts_tablet_manager.cc:1939] T 84b90b5b0a394c5b8581d8e19f51c028 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting consensus metadata
I20260924 17:43:20.618449 2776 catalog_manager.cc:4841] TS 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753): tablet 84b90b5b0a394c5b8581d8e19f51c028 (table test-table [id=4955db6ee6ca446c9b3d5067036abef2]) successfully deleted
I20260924 17:43:20.618746 2777 catalog_manager.cc:4841] TS 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963): tablet 84b90b5b0a394c5b8581d8e19f51c028 (table test-table [id=4955db6ee6ca446c9b3d5067036abef2]) successfully deleted
I20260924 17:43:20.618957 3298 tablet_bootstrap.cc:492] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: No bootstrap required, opened a new log
I20260924 17:43:20.619014 3298 ts_tablet_manager.cc:1397] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 17:43:20.619174 3298 raft_consensus.cc:348] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.619285 3298 raft_consensus.cc:374] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.619319 3298 raft_consensus.cc:729] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3614654fc5184f2e8dc47da6d498926b, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.619372 3298 consensus_queue.cc:260] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.619478 3298 ts_tablet_manager.cc:1428] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:43:20.620128 3301 tablet_bootstrap.cc:492] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: No bootstrap required, opened a new log
I20260924 17:43:20.620183 3301 ts_tablet_manager.cc:1397] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 17:43:20.620366 3301 raft_consensus.cc:348] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.620441 3301 raft_consensus.cc:374] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.620468 3301 raft_consensus.cc:729] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.620510 3301 consensus_queue.cc:260] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.620604 3301 ts_tablet_manager.cc:1428] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:20.622447 3297 tablet_bootstrap.cc:492] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: No bootstrap required, opened a new log
I20260924 17:43:20.622509 3297 ts_tablet_manager.cc:1397] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 17:43:20.622673 3297 raft_consensus.cc:348] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.622752 3297 raft_consensus.cc:374] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:20.622780 3297 raft_consensus.cc:729] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Initialized, Role: FOLLOWER
I20260924 17:43:20.622761 3345 ts_tablet_manager.cc:1918] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:20.622844 3345 log.cc:1196] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/84b90b5b0a394c5b8581d8e19f51c028
I20260924 17:43:20.622828 3297 consensus_queue.cc:260] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.622938 3297 ts_tablet_manager.cc:1428] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:20.623116 3345 ts_tablet_manager.cc:1939] T 84b90b5b0a394c5b8581d8e19f51c028 P 3614654fc5184f2e8dc47da6d498926b: Deleting consensus metadata
I20260924 17:43:20.624032 2776 catalog_manager.cc:4841] TS 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805): tablet 84b90b5b0a394c5b8581d8e19f51c028 (table test-table [id=4955db6ee6ca446c9b3d5067036abef2]) successfully deleted
I20260924 17:43:20.630980 3305 raft_consensus.cc:482] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:20.631093 3305 raft_consensus.cc:504] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.631248 3305 leader_election.cc:288] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:20.631529 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a1fec3a09a4545b2b6af6b297f1fd5" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab2435a50bb4f9b93d717668f15a07b" is_pre_election: true
I20260924 17:43:20.631562 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a1fec3a09a4545b2b6af6b297f1fd5" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b" is_pre_election: true
I20260924 17:43:20.631637 3228 raft_consensus.cc:2454] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 0.
I20260924 17:43:20.631654 2948 raft_consensus.cc:2454] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 0.
I20260924 17:43:20.631820 3021 leader_election.cc:302] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab2435a50bb4f9b93d717668f15a07b, 6cde7eaf7ba44f46999a9c6edfdcc47d; no voters:
I20260924 17:43:20.631966 3305 raft_consensus.cc:2781] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:20.632009 3305 raft_consensus.cc:482] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:20.632035 3305 raft_consensus.cc:3037] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.632905 3305 raft_consensus.cc:504] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.633076 3305 leader_election.cc:288] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 election: Requested vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:20.633337 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a1fec3a09a4545b2b6af6b297f1fd5" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab2435a50bb4f9b93d717668f15a07b"
I20260924 17:43:20.633421 3228 raft_consensus.cc:3037] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.633405 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a1fec3a09a4545b2b6af6b297f1fd5" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b"
I20260924 17:43:20.633520 2948 raft_consensus.cc:3037] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:20.634277 3228 raft_consensus.cc:2454] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 1.
I20260924 17:43:20.634344 2948 raft_consensus.cc:2454] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 1.
I20260924 17:43:20.634495 3021 leader_election.cc:302] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab2435a50bb4f9b93d717668f15a07b, 6cde7eaf7ba44f46999a9c6edfdcc47d; no voters:
I20260924 17:43:20.634577 3305 raft_consensus.cc:2781] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:20.634609 3305 raft_consensus.cc:686] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 LEADER]: Becoming Leader. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Running, Role: LEADER
I20260924 17:43:20.634652 3305 consensus_queue.cc:237] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:20.635258 2791 catalog_manager.cc:5495] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130). New cstate: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260924 17:43:20.648545 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.645438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.1.8.130:36753)
0924 17:43:20.645481 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.645483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.645492 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:20.645555 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.645556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.645566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.645637 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:20.645748 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.646509 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.647637 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.647923 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.648003 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.648165 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.648229 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.648286 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.648438 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.648457 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:20.654253 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.651186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:49307 (local address 127.1.8.130:36753)
0924 17:43:20.651291 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:20.651293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:20.651298 (+ 5us) server_negotiation.cc:407] Connection header received
0924 17:43:20.651315 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:20.651315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:20.651324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:20.651367 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:20.651519 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.652322 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.653350 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.653642 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.653733 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.653921 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.653961 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.654015 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:20.654106 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 17:43:20.654125 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 17:43:20.654253 3341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:20.651087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36753 (local address 127.1.8.130:49307)
0924 17:43:20.651134 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:20.651147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:20.651192 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:20.651396 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:20.651397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:20.651399 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:43:20.651441 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.651442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.652402 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.652403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.653263 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:20.653266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:20.653742 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:20.653743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:20.653808 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:20.653815 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:43:20.654033 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:43:20.654035 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:43:20.654091 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:43:20.654145 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":1}
I20260924 17:43:20.656658 3228 raft_consensus.cc:1261] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Refusing update from remote peer 6cde7eaf7ba44f46999a9c6edfdcc47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:20.656692 2948 raft_consensus.cc:1261] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Refusing update from remote peer 6cde7eaf7ba44f46999a9c6edfdcc47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:20.656870 3305 consensus_queue.cc:1035] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:20.656985 3333 consensus_queue.cc:1035] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:43:20.662947 3325 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 17:43:22.185725 2790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:22.185861 2790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:22.186877 2790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Sending DeleteTablet for 3 replicas of tablet 77a1fec3a09a4545b2b6af6b297f1fd5
I20260924 17:43:22.187132 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a1fec3a09a4545b2b6af6b297f1fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39648
I20260924 17:43:22.187222 3067 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a1fec3a09a4545b2b6af6b297f1fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37954
I20260924 17:43:22.187376 3373 tablet_replica.cc:330] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: stopping tablet replica
I20260924 17:43:22.187441 3373 raft_consensus.cc:2229] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:22.187480 3373 raft_consensus.cc:2258] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:22.187490 3374 tablet_replica.cc:330] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: stopping tablet replica
I20260924 17:43:22.187548 3374 raft_consensus.cc:2229] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:22.187649 3374 raft_consensus.cc:2258] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:22.187875 3373 ts_tablet_manager.cc:1905] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:22.187963 3374 ts_tablet_manager.cc:1905] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:22.188859 2928 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a1fec3a09a4545b2b6af6b297f1fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42062
I20260924 17:43:22.189563 3373 ts_tablet_manager.cc:1918] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:43:22.189617 3373 log.cc:1196] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/77a1fec3a09a4545b2b6af6b297f1fd5
I20260924 17:43:22.189864 3373 ts_tablet_manager.cc:1939] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting consensus metadata
I20260924 17:43:22.189857 3374 ts_tablet_manager.cc:1918] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:43:22.189913 3374 log.cc:1196] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/77a1fec3a09a4545b2b6af6b297f1fd5
I20260924 17:43:22.190176 3374 ts_tablet_manager.cc:1939] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting consensus metadata
I20260924 17:43:22.190523 2777 catalog_manager.cc:4841] TS 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963): tablet 77a1fec3a09a4545b2b6af6b297f1fd5 (table test-table [id=708dc891536b4cb98c0eeb89a80dcf05]) successfully deleted
I20260924 17:43:22.190536 2776 catalog_manager.cc:4841] TS 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753): tablet 77a1fec3a09a4545b2b6af6b297f1fd5 (table test-table [id=708dc891536b4cb98c0eeb89a80dcf05]) successfully deleted
I20260924 17:43:22.190874 3376 tablet_replica.cc:330] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: stopping tablet replica
I20260924 17:43:22.190935 3376 raft_consensus.cc:2229] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:22.190986 3376 raft_consensus.cc:2258] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:22.191202 3376 ts_tablet_manager.cc:1905] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:22.192469 3376 ts_tablet_manager.cc:1918] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:43:22.192519 3376 log.cc:1196] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/77a1fec3a09a4545b2b6af6b297f1fd5
I20260924 17:43:22.192744 3376 ts_tablet_manager.cc:1939] T 77a1fec3a09a4545b2b6af6b297f1fd5 P 3614654fc5184f2e8dc47da6d498926b: Deleting consensus metadata
I20260924 17:43:22.193089 2776 catalog_manager.cc:4841] TS 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805): tablet 77a1fec3a09a4545b2b6af6b297f1fd5 (table test-table [id=708dc891536b4cb98c0eeb89a80dcf05]) successfully deleted
Sep 24 17:43:22 dist-test-slave-r7hf krb5kdc[2746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271802, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:43:22 dist-test-slave-r7hf krb5kdc[2746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271802, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:22.217749 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:22.204923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.1.8.190:34169)
0924 17:43:22.205135 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:22.205137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:22.205147 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:22.205183 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:22.205183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:22.205194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:22.205286 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:22.205989 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:22.206581 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:22.207483 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:22.207758 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:22.207818 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:22.214731 (+ 6913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:22.214733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:22.214734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:22.214750 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:22.216375 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:22.216836 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:22.216837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:22.216838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:22.216904 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:22.217141 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:22.217142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:22.217142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:22.217270 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:22.217385 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:22.217527 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 17:43:22.217630 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260924 17:43:22.219542 2790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:22.219703 2790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:22.221199 3208 tablet_service.cc:1467] Processing CreateTablet for tablet 52f8f004a7244e6b94361072f099e6f8 (DEFAULT_TABLE table=test-table [id=6107754d617b4864b5963f65bba767c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:22.221215 3067 tablet_service.cc:1467] Processing CreateTablet for tablet 52f8f004a7244e6b94361072f099e6f8 (DEFAULT_TABLE table=test-table [id=6107754d617b4864b5963f65bba767c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:22.221362 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f8f004a7244e6b94361072f099e6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:22.221369 3067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f8f004a7244e6b94361072f099e6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:22.222484 3384 tablet_bootstrap.cc:492] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Bootstrap starting.
I20260924 17:43:22.222579 2928 tablet_service.cc:1467] Processing CreateTablet for tablet 52f8f004a7244e6b94361072f099e6f8 (DEFAULT_TABLE table=test-table [id=6107754d617b4864b5963f65bba767c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:22.222707 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f8f004a7244e6b94361072f099e6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:22.223066 3384 tablet_bootstrap.cc:654] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:22.223649 3384 tablet_bootstrap.cc:492] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: No bootstrap required, opened a new log
I20260924 17:43:22.223693 3384 ts_tablet_manager.cc:1397] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:22.223827 3384 raft_consensus.cc:348] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.223889 3384 raft_consensus.cc:374] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:22.223903 3384 raft_consensus.cc:729] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Initialized, Role: FOLLOWER
I20260924 17:43:22.223965 3384 consensus_queue.cc:260] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.224136 3386 tablet_bootstrap.cc:492] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Bootstrap starting.
I20260924 17:43:22.224170 3384 ts_tablet_manager.cc:1428] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:22.224351 3388 tablet_bootstrap.cc:492] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Bootstrap starting.
I20260924 17:43:22.224592 3386 tablet_bootstrap.cc:654] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:22.224768 3388 tablet_bootstrap.cc:654] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:22.225235 3386 tablet_bootstrap.cc:492] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: No bootstrap required, opened a new log
I20260924 17:43:22.225294 3386 ts_tablet_manager.cc:1397] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:22.225322 3388 tablet_bootstrap.cc:492] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: No bootstrap required, opened a new log
I20260924 17:43:22.225374 3388 ts_tablet_manager.cc:1397] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:22.225421 3386 raft_consensus.cc:348] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.225504 3386 raft_consensus.cc:374] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:22.225525 3386 raft_consensus.cc:729] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3614654fc5184f2e8dc47da6d498926b, State: Initialized, Role: FOLLOWER
I20260924 17:43:22.225478 3388 raft_consensus.cc:348] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.225548 3388 raft_consensus.cc:374] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:22.225577 3388 raft_consensus.cc:729] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Initialized, Role: FOLLOWER
I20260924 17:43:22.225570 3386 consensus_queue.cc:260] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.225663 3388 consensus_queue.cc:260] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.225762 3386 ts_tablet_manager.cc:1428] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:22.225795 3388 ts_tablet_manager.cc:1428] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:22.317749 3361 raft_consensus.cc:482] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:22.317893 3361 raft_consensus.cc:504] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.318142 3361 leader_election.cc:288] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:22.318315 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f8f004a7244e6b94361072f099e6f8" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b" is_pre_election: true
I20260924 17:43:22.318447 2948 raft_consensus.cc:2454] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 0.
I20260924 17:43:22.318367 3227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f8f004a7244e6b94361072f099e6f8" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab2435a50bb4f9b93d717668f15a07b" is_pre_election: true
I20260924 17:43:22.318511 3227 raft_consensus.cc:2454] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 0.
I20260924 17:43:22.318691 3020 leader_election.cc:302] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3614654fc5184f2e8dc47da6d498926b, 6cde7eaf7ba44f46999a9c6edfdcc47d; no voters:
I20260924 17:43:22.318783 3361 raft_consensus.cc:2781] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:22.318853 3361 raft_consensus.cc:482] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:22.318869 3361 raft_consensus.cc:3037] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:22.319607 3361 raft_consensus.cc:504] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.319746 3361 leader_election.cc:288] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 election: Requested vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963)
I20260924 17:43:22.319893 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f8f004a7244e6b94361072f099e6f8" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b"
I20260924 17:43:22.320007 2948 raft_consensus.cc:3037] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:22.319972 3227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f8f004a7244e6b94361072f099e6f8" candidate_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab2435a50bb4f9b93d717668f15a07b"
I20260924 17:43:22.320084 3227 raft_consensus.cc:3037] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:22.320729 2948 raft_consensus.cc:2454] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 1.
I20260924 17:43:22.320734 3227 raft_consensus.cc:2454] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cde7eaf7ba44f46999a9c6edfdcc47d in term 1.
I20260924 17:43:22.320926 3020 leader_election.cc:302] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3614654fc5184f2e8dc47da6d498926b, 6cde7eaf7ba44f46999a9c6edfdcc47d; no voters:
I20260924 17:43:22.321026 3361 raft_consensus.cc:2781] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:22.321075 3361 raft_consensus.cc:686] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 LEADER]: Becoming Leader. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Running, Role: LEADER
I20260924 17:43:22.321118 3361 consensus_queue.cc:237] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.321753 2791 catalog_manager.cc:5495] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130). New cstate: current_term: 1 leader_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } health_report { overall_health: UNKNOWN } } }
I20260924 17:43:22.345115 3394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:22.342801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.1.8.130:36753)
0924 17:43:22.342945 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:22.342947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:22.342956 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:22.342975 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:22.342975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:22.342985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:22.343044 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:22.343186 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:22.343686 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:22.344433 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:22.344626 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:22.344684 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:22.344835 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:22.344863 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:22.344905 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:22.344971 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 17:43:22.344990 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260924 17:43:22.347378 3227 raft_consensus.cc:1261] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Refusing update from remote peer 6cde7eaf7ba44f46999a9c6edfdcc47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:22.347398 2948 raft_consensus.cc:1261] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Refusing update from remote peer 6cde7eaf7ba44f46999a9c6edfdcc47d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:22.347600 3361 consensus_queue.cc:1035] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:22.347791 3356 consensus_queue.cc:1035] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:22.357239 2790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:22.357366 2790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:22.358065 2790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Sending DeleteTablet for 3 replicas of tablet 52f8f004a7244e6b94361072f099e6f8
I20260924 17:43:22.358355 2928 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f8f004a7244e6b94361072f099e6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42062
I20260924 17:43:22.358353 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f8f004a7244e6b94361072f099e6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39648
I20260924 17:43:22.358484 3376 tablet_replica.cc:330] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: stopping tablet replica
I20260924 17:43:22.358541 3376 raft_consensus.cc:2229] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:22.358531 3067 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f8f004a7244e6b94361072f099e6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37954
I20260924 17:43:22.358587 3376 raft_consensus.cc:2258] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:22.358605 3374 tablet_replica.cc:330] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: stopping tablet replica
I20260924 17:43:22.358580 2790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 17:43:22.358659 3373 tablet_replica.cc:330] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: stopping tablet replica
I20260924 17:43:22.358721 3373 raft_consensus.cc:2229] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:22.358726 3374 raft_consensus.cc:2229] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 LEADER]: Raft consensus shutting down.
W20260924 17:43:22.358741 2790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:22.358768 3373 raft_consensus.cc:2258] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:22.358784 3374 raft_consensus.cc:2258] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:22.359139 3376 ts_tablet_manager.cc:1905] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:22.359143 3373 ts_tablet_manager.cc:1905] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:22.360165 2928 tablet_service.cc:1467] Processing CreateTablet for tablet b3991493b80f4dbe83d9449400c6c18f (DEFAULT_TABLE table=test-table [id=e77910c796f34d28883f7d77accff74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:22.360262 2928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3991493b80f4dbe83d9449400c6c18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:22.360841 3067 tablet_service.cc:1467] Processing CreateTablet for tablet b3991493b80f4dbe83d9449400c6c18f (DEFAULT_TABLE table=test-table [id=e77910c796f34d28883f7d77accff74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:22.360896 3374 ts_tablet_manager.cc:1905] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:22.360950 3067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3991493b80f4dbe83d9449400c6c18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:22.361088 3373 ts_tablet_manager.cc:1918] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:22.361145 3376 ts_tablet_manager.cc:1918] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:22.361159 3373 log.cc:1196] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/52f8f004a7244e6b94361072f099e6f8
I20260924 17:43:22.361203 3376 log.cc:1196] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/52f8f004a7244e6b94361072f099e6f8
I20260924 17:43:22.361397 3373 ts_tablet_manager.cc:1939] T 52f8f004a7244e6b94361072f099e6f8 P 2ab2435a50bb4f9b93d717668f15a07b: Deleting consensus metadata
I20260924 17:43:22.361428 3376 ts_tablet_manager.cc:1939] T 52f8f004a7244e6b94361072f099e6f8 P 3614654fc5184f2e8dc47da6d498926b: Deleting consensus metadata
I20260924 17:43:22.361753 2776 catalog_manager.cc:4841] TS 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805): tablet 52f8f004a7244e6b94361072f099e6f8 (table test-table [id=6107754d617b4864b5963f65bba767c5]) successfully deleted
I20260924 17:43:22.362380 3388 tablet_bootstrap.cc:492] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Bootstrap starting.
I20260924 17:43:22.362632 2777 catalog_manager.cc:4841] TS 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963): tablet 52f8f004a7244e6b94361072f099e6f8 (table test-table [id=6107754d617b4864b5963f65bba767c5]) successfully deleted
I20260924 17:43:22.362659 3208 tablet_service.cc:1467] Processing CreateTablet for tablet b3991493b80f4dbe83d9449400c6c18f (DEFAULT_TABLE table=test-table [id=e77910c796f34d28883f7d77accff74e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:22.362768 3388 tablet_bootstrap.cc:654] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:22.362799 3208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3991493b80f4dbe83d9449400c6c18f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:22.363054 3374 ts_tablet_manager.cc:1918] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:43:22.363102 3374 log.cc:1196] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/52f8f004a7244e6b94361072f099e6f8
I20260924 17:43:22.363318 3374 ts_tablet_manager.cc:1939] T 52f8f004a7244e6b94361072f099e6f8 P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting consensus metadata
I20260924 17:43:22.363433 3388 tablet_bootstrap.cc:492] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: No bootstrap required, opened a new log
I20260924 17:43:22.363480 3388 ts_tablet_manager.cc:1397] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:22.363615 3388 raft_consensus.cc:348] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.363679 3388 raft_consensus.cc:374] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:22.363698 3388 raft_consensus.cc:729] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cde7eaf7ba44f46999a9c6edfdcc47d, State: Initialized, Role: FOLLOWER
I20260924 17:43:22.363749 3388 consensus_queue.cc:260] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.363848 3388 ts_tablet_manager.cc:1428] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:22.363993 3384 tablet_bootstrap.cc:492] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Bootstrap starting.
I20260924 17:43:22.364180 3386 tablet_bootstrap.cc:492] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b: Bootstrap starting.
I20260924 17:43:22.364346 3384 tablet_bootstrap.cc:654] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:22.364567 3386 tablet_bootstrap.cc:654] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:22.364655 2776 catalog_manager.cc:4841] TS 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753): tablet 52f8f004a7244e6b94361072f099e6f8 (table test-table [id=6107754d617b4864b5963f65bba767c5]) successfully deleted
I20260924 17:43:22.365095 3384 tablet_bootstrap.cc:492] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: No bootstrap required, opened a new log
I20260924 17:43:22.365149 3384 ts_tablet_manager.cc:1397] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:22.365265 3384 raft_consensus.cc:348] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.365343 3384 raft_consensus.cc:374] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:22.365371 3384 raft_consensus.cc:729] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Initialized, Role: FOLLOWER
I20260924 17:43:22.365420 3384 consensus_queue.cc:260] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.365511 3386 tablet_bootstrap.cc:492] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b: No bootstrap required, opened a new log
I20260924 17:43:22.365525 3384 ts_tablet_manager.cc:1428] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:22.365568 3386 ts_tablet_manager.cc:1397] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:22.365729 3386 raft_consensus.cc:348] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.365820 3386 raft_consensus.cc:374] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:22.365845 3386 raft_consensus.cc:729] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3614654fc5184f2e8dc47da6d498926b, State: Initialized, Role: FOLLOWER
I20260924 17:43:22.365890 3386 consensus_queue.cc:260] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.366000 3386 ts_tablet_manager.cc:1428] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:22.571403 3387 raft_consensus.cc:482] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:22.571516 3387 raft_consensus.cc:504] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.571684 3387 leader_election.cc:288] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753)
I20260924 17:43:22.572027 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3991493b80f4dbe83d9449400c6c18f" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b" is_pre_election: true
I20260924 17:43:22.572127 2948 raft_consensus.cc:2454] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 0.
I20260924 17:43:22.572072 3088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3991493b80f4dbe83d9449400c6c18f" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" is_pre_election: true
I20260924 17:43:22.572220 3088 raft_consensus.cc:2454] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 0.
I20260924 17:43:22.572324 3160 leader_election.cc:302] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab2435a50bb4f9b93d717668f15a07b, 3614654fc5184f2e8dc47da6d498926b; no voters:
I20260924 17:43:22.572429 3387 raft_consensus.cc:2781] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:22.572470 3387 raft_consensus.cc:482] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:22.572494 3387 raft_consensus.cc:3037] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:22.573182 3387 raft_consensus.cc:504] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.573328 3387 leader_election.cc:288] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 election: Requested vote from peers 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805), 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753)
I20260924 17:43:22.573509 2948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3991493b80f4dbe83d9449400c6c18f" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3614654fc5184f2e8dc47da6d498926b"
I20260924 17:43:22.573498 3088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3991493b80f4dbe83d9449400c6c18f" candidate_uuid: "2ab2435a50bb4f9b93d717668f15a07b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d"
I20260924 17:43:22.573585 2948 raft_consensus.cc:3037] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:22.573587 3088 raft_consensus.cc:3037] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:22.574198 2948 raft_consensus.cc:2454] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 1.
I20260924 17:43:22.574378 3160 leader_election.cc:302] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ab2435a50bb4f9b93d717668f15a07b, 3614654fc5184f2e8dc47da6d498926b; no voters:
I20260924 17:43:22.574461 3088 raft_consensus.cc:2454] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ab2435a50bb4f9b93d717668f15a07b in term 1.
I20260924 17:43:22.574501 3387 raft_consensus.cc:2781] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:22.574620 3387 raft_consensus.cc:686] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 1 LEADER]: Becoming Leader. State: Replica: 2ab2435a50bb4f9b93d717668f15a07b, State: Running, Role: LEADER
I20260924 17:43:22.574676 3387 consensus_queue.cc:237] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } }
I20260924 17:43:22.575371 2791 catalog_manager.cc:5495] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131). New cstate: current_term: 1 leader_uuid: "2ab2435a50bb4f9b93d717668f15a07b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab2435a50bb4f9b93d717668f15a07b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 37963 } health_report { overall_health: HEALTHY } } }
I20260924 17:43:22.596535 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:22.593898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.1.8.131:37963)
0924 17:43:22.594069 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:22.594073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:22.594086 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:43:22.594105 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:22.594105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:22.594116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:22.594180 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:22.594284 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:22.594786 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:22.595585 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:22.595841 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:22.595903 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:22.596201 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:22.596242 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:22.596287 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:22.596372 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 17:43:22.596394 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":122,"threads_started":1}
I20260924 17:43:22.599207 3088 raft_consensus.cc:1261] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Refusing update from remote peer 2ab2435a50bb4f9b93d717668f15a07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:22.599207 2948 raft_consensus.cc:1261] T b3991493b80f4dbe83d9449400c6c18f P 3614654fc5184f2e8dc47da6d498926b [term 1 FOLLOWER]: Refusing update from remote peer 2ab2435a50bb4f9b93d717668f15a07b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:22.599442 3387 consensus_queue.cc:1035] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3614654fc5184f2e8dc47da6d498926b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:22.599574 3401 consensus_queue.cc:1035] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cde7eaf7ba44f46999a9c6edfdcc47d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:43:22.604120 3357 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 17:43:24.122536 2791 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:24.122660 2791 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:24.123631 2791 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcefefb3d218494e90263abdcfa67b32: Sending DeleteTablet for 3 replicas of tablet b3991493b80f4dbe83d9449400c6c18f
I20260924 17:43:24.123911 3208 tablet_service.cc:1514] Processing DeleteTablet for tablet b3991493b80f4dbe83d9449400c6c18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:24 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39648
I20260924 17:43:24.123945 3067 tablet_service.cc:1514] Processing DeleteTablet for tablet b3991493b80f4dbe83d9449400c6c18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:24 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37954
I20260924 17:43:24.123992 2928 tablet_service.cc:1514] Processing DeleteTablet for tablet b3991493b80f4dbe83d9449400c6c18f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:24 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42062
I20260924 17:43:24.124146 3429 tablet_replica.cc:330] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: stopping tablet replica
I20260924 17:43:24.124202 3429 raft_consensus.cc:2229] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:24.124193 3430 tablet_replica.cc:330] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: stopping tablet replica
I20260924 17:43:24.124264 3430 raft_consensus.cc:2229] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:24.124303 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 2863
I20260924 17:43:24.124326 3429 raft_consensus.cc:2258] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:24.125145 3430 raft_consensus.cc:2258] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:24.125217 3429 ts_tablet_manager.cc:1905] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:24.125393 3430 ts_tablet_manager.cc:1905] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:24.127120 3430 ts_tablet_manager.cc:1918] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:43:24.127177 3430 log.cc:1196] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/b3991493b80f4dbe83d9449400c6c18f
I20260924 17:43:24.127395 3430 ts_tablet_manager.cc:1939] T b3991493b80f4dbe83d9449400c6c18f P 6cde7eaf7ba44f46999a9c6edfdcc47d: Deleting consensus metadata
I20260924 17:43:24.127384 3429 ts_tablet_manager.cc:1918] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 17:43:24.127449 3429 log.cc:1196] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/b3991493b80f4dbe83d9449400c6c18f
I20260924 17:43:24.127756 3429 ts_tablet_manager.cc:1939] T b3991493b80f4dbe83d9449400c6c18f P 2ab2435a50bb4f9b93d717668f15a07b: Deleting consensus metadata
I20260924 17:43:24.127892 2776 catalog_manager.cc:4841] TS 6cde7eaf7ba44f46999a9c6edfdcc47d (127.1.8.130:36753): tablet b3991493b80f4dbe83d9449400c6c18f (table test-table [id=e77910c796f34d28883f7d77accff74e]) successfully deleted
I20260924 17:43:24.128109 2777 catalog_manager.cc:4841] TS 2ab2435a50bb4f9b93d717668f15a07b (127.1.8.131:37963): tablet b3991493b80f4dbe83d9449400c6c18f (table test-table [id=e77910c796f34d28883f7d77accff74e]) successfully deleted
W20260924 17:43:24.131079 2776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:44805 (error 108)
I20260924 17:43:24.131314 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3003
I20260924 17:43:24.133384 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:24.133155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44805 (local address 127.0.0.1:42074)
0924 17:43:24.133293 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:24.133347 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:44805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
W20260924 17:43:24.133513 2776 catalog_manager.cc:4584] TS 3614654fc5184f2e8dc47da6d498926b (127.1.8.129:44805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3991493b80f4dbe83d9449400c6c18f: Network error: Client connection negotiation failed: client connection to 127.1.8.129:44805: connect: Connection refused (error 111)
I20260924 17:43:24.137280 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3143
I20260924 17:43:24.143111 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 2759
2026-09-24T17:43:24Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5898 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:24 dist-test-slave-r7hf krb5kdc[3436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:24 dist-test-slave-r7hf krb5kdc[3436](info): set up 2 sockets
Sep 24 17:43:24 dist-test-slave-r7hf krb5kdc[3436](info): commencing operation
krb5kdc: starting...
W20260924 17:43:25.250303 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:25.378101 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44973
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:25.460980 3449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:25.461213 3449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:25.461236 3449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:25.462574 3449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:25.462636 3449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:25.462653 3449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:25.462666 3449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:25.462679 3449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:25.464218 3449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:25.464545 3449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:25.464809 3449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:25.469947 3454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:25.469947 3457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:25.469976 3455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:25.470278 3449 server_base.cc:1034] running on GCE node
I20260924 17:43:25.470559 3449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:25.470861 3449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:25.472002 3449 hybrid_clock.cc:648] HybridClock initialized: now 1790271805471988 us; error 27 us; skew 500 ppm
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:25.474599 3449 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:25.475088 3449 webserver.cc:460] Webserver started at http://127.1.8.190:35029/ using document root <none> and password file <none>
I20260924 17:43:25.475342 3449 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:25.475404 3449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:25.475540 3449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:25.476722 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "5a60a8b851324247a5cf12e0972cd17b"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "728aba2a823a9e442fc5c176d20565a3"
server_key_iv: "b84b042f4c30396f83b7541fe9406ae8"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.477097 3449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "5a60a8b851324247a5cf12e0972cd17b"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "728aba2a823a9e442fc5c176d20565a3"
server_key_iv: "b84b042f4c30396f83b7541fe9406ae8"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.478529 3449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 17:43:25.479357 3464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.479584 3449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:25.479698 3449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "5a60a8b851324247a5cf12e0972cd17b"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "728aba2a823a9e442fc5c176d20565a3"
server_key_iv: "b84b042f4c30396f83b7541fe9406ae8"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.479781 3449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:25.489881 3449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:25.492360 3449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:25.492573 3449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:25.496858 3449 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:44973
I20260924 17:43:25.496892 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:44973 every 8 connection(s)
I20260924 17:43:25.497280 3449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:25.498235 3517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:25.500802 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b: Bootstrap starting.
I20260924 17:43:25.501492 3517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:25.501842 3517 log.cc:824] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:25.502705 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b: No bootstrap required, opened a new log
I20260924 17:43:25.503258 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3449
I20260924 17:43:25.503371 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:25.503538 1058 external_mini_cluster.cc:1428] Setting key 58a09000a810b46e05efeb5cf82f4f89
I20260924 17:43:25.505280 3517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a60a8b851324247a5cf12e0972cd17b" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44973 } }
I20260924 17:43:25.505481 3517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:25.505522 3517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a60a8b851324247a5cf12e0972cd17b, State: Initialized, Role: FOLLOWER
I20260924 17:43:25.505702 3517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a60a8b851324247a5cf12e0972cd17b" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44973 } }
I20260924 17:43:25.505793 3517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:25.505820 3517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:25.505882 3517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:25.507341 3517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a60a8b851324247a5cf12e0972cd17b" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44973 } }
I20260924 17:43:25.507462 3517 leader_election.cc:302] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a60a8b851324247a5cf12e0972cd17b; no voters:
I20260924 17:43:25.507645 3517 leader_election.cc:288] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:25.507722 3522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:25.507908 3517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:25.508013 3522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [term 1 LEADER]: Becoming Leader. State: Replica: 5a60a8b851324247a5cf12e0972cd17b, State: Running, Role: LEADER
I20260924 17:43:25.508157 3522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a60a8b851324247a5cf12e0972cd17b" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44973 } }
I20260924 17:43:25.508574 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a60a8b851324247a5cf12e0972cd17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a60a8b851324247a5cf12e0972cd17b" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44973 } } }
I20260924 17:43:25.508656 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:25.508684 3524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a60a8b851324247a5cf12e0972cd17b. Latest consensus state: current_term: 1 leader_uuid: "5a60a8b851324247a5cf12e0972cd17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a60a8b851324247a5cf12e0972cd17b" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44973 } } }
I20260924 17:43:25.508908 3524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:25.509274 3529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:25.509807 3529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:25.511636 3529 catalog_manager.cc:1347] Generated new cluster ID: 200669ebd4b1496795bd7a289afc135f
I20260924 17:43:25.511694 3529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:25.516603 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:25.504235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.1.8.190:44973)
0924 17:43:25.504504 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:25.504515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:25.504579 (+ 64us) server_negotiation.cc:407] Connection header received
0924 17:43:25.505230 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:25.505240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:25.505316 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:25.505485 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:25.506330 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.508035 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.509234 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.509915 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.510110 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:25.512918 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:25.512922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:25.512922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:25.512935 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:25.514851 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:25.515238 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:25.515241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:25.515241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:25.515347 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:25.515573 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:25.515574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:25.515575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:25.515829 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:25.516005 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:25.516082 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 17:43:25.516241 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":125,"threads_started":1}
I20260924 17:43:25.521900 3529 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:25.522795 3529 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:25.532286 3529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a60a8b851324247a5cf12e0972cd17b: Generated new TSK 0
I20260924 17:43:25.532521 3529 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:25.566511 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44973
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:25.666316 3543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:25.666664 3543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:25.666741 3543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:25.668081 3543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:25.668169 3543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:25.668221 3543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:25.669874 3543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:25.670198 3543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:25.670539 3543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:25.676088 3551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:25.676134 3548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:25.676270 3543 server_base.cc:1034] running on GCE node
W20260924 17:43:25.676183 3549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:25.676550 3543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:25.676878 3543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:25.678128 3543 hybrid_clock.cc:648] HybridClock initialized: now 1790271805678111 us; error 28 us; skew 500 ppm
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:25.680905 3543 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:25.681408 3543 webserver.cc:460] Webserver started at http://127.1.8.129:37375/ using document root <none> and password file <none>
I20260924 17:43:25.681668 3543 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:25.681748 3543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:25.681900 3543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:25.683167 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "418797407c944f799fa5c8a23fd7fc41"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "d9ad2c7334b0c9596721d5c66f0672fd"
server_key_iv: "2c96a3ecf1cc2277187eebfe21fea35b"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.683543 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "418797407c944f799fa5c8a23fd7fc41"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "d9ad2c7334b0c9596721d5c66f0672fd"
server_key_iv: "2c96a3ecf1cc2277187eebfe21fea35b"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.685025 3543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:25.685974 3558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.686199 3543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:25.686324 3543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "418797407c944f799fa5c8a23fd7fc41"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "d9ad2c7334b0c9596721d5c66f0672fd"
server_key_iv: "2c96a3ecf1cc2277187eebfe21fea35b"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.686430 3543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:25.704103 3543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:25.707021 3543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:25.707154 3543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:25.707439 3543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:25.707787 3543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:25.707845 3543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.707890 3543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:25.707911 3543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.713933 3543 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:41019
I20260924 17:43:25.713979 3671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:41019 every 8 connection(s)
I20260924 17:43:25.714362 3543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:43:25.716049 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3543
I20260924 17:43:25.716142 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:25.716320 1058 external_mini_cluster.cc:1428] Setting key f38706591e9ae3734d0bffec452c58d7
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:43:25.726125 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:25.715192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:32947 (local address 127.1.8.190:44973)
0924 17:43:25.715274 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:25.715277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:25.715529 (+ 252us) server_negotiation.cc:407] Connection header received
0924 17:43:25.716078 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:25.716080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:25.716093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:25.716170 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:25.717575 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.718431 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.720110 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.720314 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.720371 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:25.723033 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:25.723034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:25.723035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:25.723044 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:25.724729 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:25.725129 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:25.725131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:25.725131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:25.725188 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:25.725436 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:25.725437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:25.725437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:25.725604 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:25.725693 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:25.725974 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 17:43:25.726049 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 17:43:25.726286 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:25.715184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44973 (local address 127.1.8.129:32947)
0924 17:43:25.715398 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:25.715438 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:25.715960 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:25.716330 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:25.716334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:25.717223 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:25.717472 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:25.717479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.718566 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.718568 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:25.720016 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:25.720019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.720405 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.720406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:25.720564 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:25.721193 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:25.721205 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:25.722932 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:25.724819 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:25.724824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:25.724826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:25.725045 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:25.725266 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:25.725266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:25.725267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:25.725367 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:25.725689 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:25.725694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:25.725838 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 17:43:25.725930 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 17:43:25.726057 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":112,"threads_started":1}
I20260924 17:43:25.726889 3672 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44973
I20260924 17:43:25.726989 3672 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:25.727205 3672 heartbeater.cc:511] Master 127.1.8.190:44973 requested a full tablet report, sending...
I20260924 17:43:25.727730 3481 ts_manager.cc:194] Registered new tserver with Master: 418797407c944f799fa5c8a23fd7fc41 (127.1.8.129:41019)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
I20260924 17:43:25.728467 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:32947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:25.746942 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44973
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:25.831144 3677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:25.831377 3677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:25.831408 3677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:25.832674 3677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:25.832724 3677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:25.832753 3677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:25.834291 3677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:25.834601 3677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:25.834873 3677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:25.840324 3685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:25.840365 3683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:25.840386 3677 server_base.cc:1034] running on GCE node
W20260924 17:43:25.840365 3682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:25.840783 3677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:25.841104 3677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:25.842271 3677 hybrid_clock.cc:648] HybridClock initialized: now 1790271805842243 us; error 39 us; skew 500 ppm
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:25.844973 3677 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:25.845448 3677 webserver.cc:460] Webserver started at http://127.1.8.130:36201/ using document root <none> and password file <none>
I20260924 17:43:25.845679 3677 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:25.845731 3677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:25.845902 3677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:25.847082 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "f7d97cf96ce24e22986c0927954c4b4a"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "989d7d454dfd98d22951269211c7ac4a"
server_key_iv: "761bd7cfce7ca2c777f8f7a60d54173a"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.847465 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "f7d97cf96ce24e22986c0927954c4b4a"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "989d7d454dfd98d22951269211c7ac4a"
server_key_iv: "761bd7cfce7ca2c777f8f7a60d54173a"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.848863 3677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:25.849685 3692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.849943 3677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:25.850020 3677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "f7d97cf96ce24e22986c0927954c4b4a"
format_stamp: "Formatted at 2026-09-24 17:43:25 on dist-test-slave-r7hf"
server_key: "989d7d454dfd98d22951269211c7ac4a"
server_key_iv: "761bd7cfce7ca2c777f8f7a60d54173a"
server_key_version: "encryptionkey@0"
I20260924 17:43:25.850116 3677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:25.862360 3677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:25.864837 3677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:25.864992 3677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:25.865257 3677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:25.865641 3677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:25.865677 3677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.865746 3677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:25.865769 3677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:25.871986 3677 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:44617
I20260924 17:43:25.872025 3805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:44617 every 8 connection(s)
I20260924 17:43:25.872460 3677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:43:25 dist-test-slave-r7hf krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271805, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:25.882098 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3677
I20260924 17:43:25.882201 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:25.882483 1058 external_mini_cluster.cc:1428] Setting key b2b7576f67d7b2f8037b0cb83bed8660
I20260924 17:43:25.882862 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:25.873216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33145 (local address 127.1.8.190:44973)
0924 17:43:25.873274 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:25.873277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:25.873508 (+ 231us) server_negotiation.cc:407] Connection header received
0924 17:43:25.874188 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:25.874190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:25.874203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:25.874249 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:25.875380 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.876011 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.877130 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.877347 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.877434 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:25.879777 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:25.879779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:25.879779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:25.879788 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:25.881088 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:25.881538 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:25.881539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:25.881540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:25.881608 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:25.881905 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:25.881906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:25.881906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:25.882064 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:25.882140 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:25.882650 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 17:43:25.882749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 17:43:25.883055 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:25.873223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44973 (local address 127.1.8.130:33145)
0924 17:43:25.873372 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:25.873411 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:25.874071 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:25.874359 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:25.874362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:25.875071 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:25.875271 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:25.875276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.876107 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.876108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:25.877019 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:25.877021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:25.877438 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:25.877439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:25.877579 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:25.878093 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:25.878103 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:25.879663 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:25.881200 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:25.881207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:25.881210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:25.881440 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:25.881711 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:25.881712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:25.881713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:25.881827 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:25.882197 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:25.882204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:25.882458 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 17:43:25.882595 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 17:43:25.882814 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260924 17:43:25.883714 3806 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44973
I20260924 17:43:25.883833 3806 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:25.884039 3806 heartbeater.cc:511] Master 127.1.8.190:44973 requested a full tablet report, sending...
I20260924 17:43:25.884460 3481 ts_manager.cc:194] Registered new tserver with Master: f7d97cf96ce24e22986c0927954c4b4a (127.1.8.130:44617)
I20260924 17:43:25.884905 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:33145
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:25.910591 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44973
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:25.993606 3811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:25.993918 3811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:25.993966 3811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:25.995294 3811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:25.995375 3811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:25.995424 3811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:25.996968 3811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:25.997272 3811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:25.997522 3811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:26.003018 3819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:26.003018 3816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:26.003202 3811 server_base.cc:1034] running on GCE node
W20260924 17:43:26.003058 3817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:26.003509 3811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:26.003804 3811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:26.005005 3811 hybrid_clock.cc:648] HybridClock initialized: now 1790271806004973 us; error 43 us; skew 500 ppm
Sep 24 17:43:26 dist-test-slave-r7hf krb5kdc[3436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:26.007805 3811 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:26.008443 3811 webserver.cc:460] Webserver started at http://127.1.8.131:46505/ using document root <none> and password file <none>
I20260924 17:43:26.008778 3811 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:26.008853 3811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:26.009006 3811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:26.010277 3811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "3309f1db35e64a99944113886aaf147d"
format_stamp: "Formatted at 2026-09-24 17:43:26 on dist-test-slave-r7hf"
server_key: "91a7ed0620d68e05587058fd1b43f73b"
server_key_iv: "15114e617f91a6857e9992aa4b0a5e3b"
server_key_version: "encryptionkey@0"
I20260924 17:43:26.010751 3811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "3309f1db35e64a99944113886aaf147d"
format_stamp: "Formatted at 2026-09-24 17:43:26 on dist-test-slave-r7hf"
server_key: "91a7ed0620d68e05587058fd1b43f73b"
server_key_iv: "15114e617f91a6857e9992aa4b0a5e3b"
server_key_version: "encryptionkey@0"
I20260924 17:43:26.012156 3811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:26.012933 3826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:26.013141 3811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:26.013221 3811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "3309f1db35e64a99944113886aaf147d"
format_stamp: "Formatted at 2026-09-24 17:43:26 on dist-test-slave-r7hf"
server_key: "91a7ed0620d68e05587058fd1b43f73b"
server_key_iv: "15114e617f91a6857e9992aa4b0a5e3b"
server_key_version: "encryptionkey@0"
I20260924 17:43:26.013341 3811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:26.028306 3811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:26.030297 3811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:26.030503 3811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:26.030767 3811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:26.031091 3811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:26.031150 3811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:26.031205 3811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:26.031232 3811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:26.037223 3811 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:44203
I20260924 17:43:26.037312 3939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:44203 every 8 connection(s)
I20260924 17:43:26.037652 3811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:26 dist-test-slave-r7hf krb5kdc[3436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:26.045797 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3811
I20260924 17:43:26.045869 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:26.046061 1058 external_mini_cluster.cc:1428] Setting key bb8dc72c0afca42f725a72d73169dd11
I20260924 17:43:26.047408 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:26.038471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:37385 (local address 127.1.8.190:44973)
0924 17:43:26.038540 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:26.038542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:26.038766 (+ 224us) server_negotiation.cc:407] Connection header received
0924 17:43:26.039254 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:26.039255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:26.039264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:26.039330 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:26.040254 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:26.040762 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:26.041732 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:26.041907 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:26.041957 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:26.044260 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:26.044261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:26.044261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:26.044279 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:26.045697 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:26.046183 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:26.046184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:26.046185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:26.046265 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:26.046656 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:26.046657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:26.046657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:26.046809 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:26.046874 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:26.047273 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 17:43:26.047346 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:43:26.047533 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:26.038452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44973 (local address 127.1.8.131:37385)
0924 17:43:26.038625 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:26.038669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:26.039166 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:26.039388 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:26.039392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:26.039968 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:26.040167 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:26.040172 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:26.040858 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:26.040859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:26.041657 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:26.041658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:26.041980 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:26.041981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:26.042075 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:26.042684 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:26.042692 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:26.044143 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:26.045818 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:26.045825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:26.045828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:26.046118 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:26.046363 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:26.046364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:26.046365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:26.046558 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:26.046880 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:26.046886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:26.047081 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 17:43:26.047188 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 17:43:26.047321 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260924 17:43:26.047969 3940 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44973
I20260924 17:43:26.048056 3940 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:26.048269 3940 heartbeater.cc:511] Master 127.1.8.190:44973 requested a full tablet report, sending...
I20260924 17:43:26.048633 3481 ts_manager.cc:194] Registered new tserver with Master: 3309f1db35e64a99944113886aaf147d (127.1.8.131:44203)
I20260924 17:43:26.049023 3481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:37385
I20260924 17:43:26.058264 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 17:43:26.069979 3521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:26.068944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.1.8.190:44973)
0924 17:43:26.069009 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:26.069012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:26.069141 (+ 129us) server_negotiation.cc:407] Connection header received
0924 17:43:26.069229 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:26.069230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:26.069248 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:26.069314 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:26.069897 (+ 583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54100: BlockingRecv error: recv got EOF from 127.0.0.1:54100 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:26.071033 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3543
I20260924 17:43:26.075459 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3677
I20260924 17:43:26.079735 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3811
I20260924 17:43:26.084120 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3449
2026-09-24T17:43:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1939 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:26 dist-test-slave-r7hf krb5kdc[3952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:26 dist-test-slave-r7hf krb5kdc[3952](info): set up 2 sockets
Sep 24 17:43:26 dist-test-slave-r7hf krb5kdc[3952](info): commencing operation
krb5kdc: starting...
W20260924 17:43:27.134300 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:27 dist-test-slave-r7hf krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:27.257925 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45411
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:27.343883 3965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:27.344130 3965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:27.344156 3965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:27.345443 3965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:27.345503 3965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:27.345523 3965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:27.345575 3965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:27.345613 3965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:27.347353 3965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:27.347662 3965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:27.347918 3965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:27.353209 3971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:27.353214 3970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:27.353214 3973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:27.353627 3965 server_base.cc:1034] running on GCE node
I20260924 17:43:27.353914 3965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:27.354259 3965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:27.355429 3965 hybrid_clock.cc:648] HybridClock initialized: now 1790271807355397 us; error 46 us; skew 500 ppm
Sep 24 17:43:27 dist-test-slave-r7hf krb5kdc[3952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271807, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:27.366858 3965 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:27.367533 3965 webserver.cc:460] Webserver started at http://127.1.8.190:35317/ using document root <none> and password file <none>
I20260924 17:43:27.367798 3965 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:27.367885 3965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:27.368073 3965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:27.369293 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4"
format_stamp: "Formatted at 2026-09-24 17:43:27 on dist-test-slave-r7hf"
server_key: "966bc4a249f68500e45500d22087f67c"
server_key_iv: "7d1363b7d059fa5f93d3c1f46a55ee10"
server_key_version: "encryptionkey@0"
I20260924 17:43:27.369714 3965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4"
format_stamp: "Formatted at 2026-09-24 17:43:27 on dist-test-slave-r7hf"
server_key: "966bc4a249f68500e45500d22087f67c"
server_key_iv: "7d1363b7d059fa5f93d3c1f46a55ee10"
server_key_version: "encryptionkey@0"
I20260924 17:43:27.371372 3965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 17:43:27.372314 3980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:27.372512 3965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:27.372692 3965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4"
format_stamp: "Formatted at 2026-09-24 17:43:27 on dist-test-slave-r7hf"
server_key: "966bc4a249f68500e45500d22087f67c"
server_key_iv: "7d1363b7d059fa5f93d3c1f46a55ee10"
server_key_version: "encryptionkey@0"
I20260924 17:43:27.372771 3965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:27.387786 3965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:27.389828 3965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:27.389991 3965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:27.394097 3965 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:45411
I20260924 17:43:27.394158 4032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:45411 every 8 connection(s)
I20260924 17:43:27.394555 3965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:27.395514 4033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:27.397861 4033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4: Bootstrap starting.
I20260924 17:43:27.398553 4033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:27.398862 4033 log.cc:824] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:27.399572 4033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4: No bootstrap required, opened a new log
I20260924 17:43:27.401162 4033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45411 } }
I20260924 17:43:27.401330 4033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:27.401355 4033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab7f3e47bb043c098b5b7768bc4e8d4, State: Initialized, Role: FOLLOWER
I20260924 17:43:27.401516 4033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45411 } }
I20260924 17:43:27.401597 4033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:27.401620 4033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:27.401650 4033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:27.402654 4033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45411 } }
I20260924 17:43:27.402750 4033 leader_election.cc:302] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ab7f3e47bb043c098b5b7768bc4e8d4; no voters:
I20260924 17:43:27.402987 4033 leader_election.cc:288] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:27.403133 4036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:27.403290 4033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:27.403371 4036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [term 1 LEADER]: Becoming Leader. State: Replica: 5ab7f3e47bb043c098b5b7768bc4e8d4, State: Running, Role: LEADER
I20260924 17:43:27.403474 4036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45411 } }
I20260924 17:43:27.403508 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 3965
I20260924 17:43:27.403594 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:27.403792 1058 external_mini_cluster.cc:1428] Setting key bc41ee8863dcaf2ace7f2af80aaddc56
I20260924 17:43:27.403878 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ab7f3e47bb043c098b5b7768bc4e8d4. Latest consensus state: current_term: 1 leader_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45411 } } }
I20260924 17:43:27.403944 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:27.404181 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ab7f3e47bb043c098b5b7768bc4e8d4" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45411 } } }
I20260924 17:43:27.404290 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4 [sys.catalog]: This master's current role is: LEADER
W20260924 17:43:27.405610 4053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:27.405675 4053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:27.405747 4044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:27.405926 4044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:27.407581 4044 catalog_manager.cc:1347] Generated new cluster ID: 3e53b11f3c63402db19107887a47fad5
I20260924 17:43:27.407635 4044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:43:27 dist-test-slave-r7hf krb5kdc[3952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:27.416011 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:27.404332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.1.8.190:45411)
0924 17:43:27.404540 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:27.404549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:27.404573 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:43:27.404719 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:27.404726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:27.404797 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:27.405813 (+ 1016us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:27.405916 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:27.407183 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:27.408286 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:27.408940 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:27.409047 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:27.411971 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:27.411977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:27.411977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:27.411993 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:27.414160 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:27.414616 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:27.414620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:27.414621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:27.414769 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:27.414982 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:27.414983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:27.414984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:27.415351 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:27.415536 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:27.415610 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 17:43:27.415773 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260924 17:43:27.417130 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:27.422823 4044 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:27.423590 4044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:27.426980 4044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ab7f3e47bb043c098b5b7768bc4e8d4: Generated new TSK 0
I20260924 17:43:27.427178 4044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:27.435797 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:27.431402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.1.8.190:45411)
0924 17:43:27.431466 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:27.431468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:27.431663 (+ 195us) server_negotiation.cc:407] Connection header received
0924 17:43:27.431719 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:27.431720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:27.431733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:27.431868 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:27.432287 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:27.433103 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:27.433921 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:27.434265 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:27.434337 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:27.435302 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:27.435304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:27.435304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:27.435505 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:27.435523 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:27.435529 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:27.435580 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:27.435667 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 17:43:27.435699 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:43:27.438205 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 3965
2026-09-24T17:43:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1351 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:27 dist-test-slave-r7hf krb5kdc[4066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:27 dist-test-slave-r7hf krb5kdc[4066](info): set up 2 sockets
Sep 24 17:43:27 dist-test-slave-r7hf krb5kdc[4066](info): commencing operation
krb5kdc: starting...
W20260924 17:43:28.490355 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:28 dist-test-slave-r7hf krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:28.617851 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44993
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:28.706257 4079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:28.706602 4079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:28.706633 4079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:28.707871 4079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:28.707928 4079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:28.707943 4079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:28.707957 4079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:28.707969 4079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:28.710041 4079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:28.710304 4079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:28.710623 4079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:28.715883 4087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:28.715991 4085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:28.715883 4084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:28.716156 4079 server_base.cc:1034] running on GCE node
I20260924 17:43:28.716535 4079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:28.716838 4079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:28.717988 4079 hybrid_clock.cc:648] HybridClock initialized: now 1790271808717965 us; error 51 us; skew 500 ppm
Sep 24 17:43:28 dist-test-slave-r7hf krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:28.720882 4079 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:28.721428 4079 webserver.cc:460] Webserver started at http://127.1.8.190:33609/ using document root <none> and password file <none>
I20260924 17:43:28.721670 4079 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:28.721717 4079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:28.721904 4079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:28.723165 4079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "8e3e743bcec647e0bc9eeeef33adccf9"
format_stamp: "Formatted at 2026-09-24 17:43:28 on dist-test-slave-r7hf"
server_key: "e1445e5562600eba98982a7cb415832f"
server_key_iv: "84b61ccf26c58a3becea70de456fe9a9"
server_key_version: "encryptionkey@0"
I20260924 17:43:28.723637 4079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "8e3e743bcec647e0bc9eeeef33adccf9"
format_stamp: "Formatted at 2026-09-24 17:43:28 on dist-test-slave-r7hf"
server_key: "e1445e5562600eba98982a7cb415832f"
server_key_iv: "84b61ccf26c58a3becea70de456fe9a9"
server_key_version: "encryptionkey@0"
I20260924 17:43:28.725108 4079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:28.725948 4094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:28.726132 4079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:28.726244 4079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "8e3e743bcec647e0bc9eeeef33adccf9"
format_stamp: "Formatted at 2026-09-24 17:43:28 on dist-test-slave-r7hf"
server_key: "e1445e5562600eba98982a7cb415832f"
server_key_iv: "84b61ccf26c58a3becea70de456fe9a9"
server_key_version: "encryptionkey@0"
I20260924 17:43:28.726323 4079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:28.735666 4079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:28.737754 4079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:28.737936 4079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:28.742209 4079 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:44993
I20260924 17:43:28.742244 4146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:44993 every 8 connection(s)
I20260924 17:43:28.742723 4079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:28.743204 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4079
I20260924 17:43:28.743315 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:28.743505 1058 external_mini_cluster.cc:1428] Setting key cb6e747f484a2490b2b200569e3fa905
I20260924 17:43:28.744010 4147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:28.747534 4147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9: Bootstrap starting.
I20260924 17:43:28.748224 4147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:28.748554 4147 log.cc:824] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:28.749392 4147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9: No bootstrap required, opened a new log
Sep 24 17:43:28 dist-test-slave-r7hf krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:28.751361 4147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44993 } }
I20260924 17:43:28.751526 4147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:28.751592 4147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3e743bcec647e0bc9eeeef33adccf9, State: Initialized, Role: FOLLOWER
I20260924 17:43:28.751777 4147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44993 } }
I20260924 17:43:28.751888 4147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:28.751948 4147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:28.752004 4147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:28.752635 4147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44993 } }
I20260924 17:43:28.752756 4147 leader_election.cc:302] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e3e743bcec647e0bc9eeeef33adccf9; no voters:
I20260924 17:43:28.752970 4147 leader_election.cc:288] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:28.753008 4152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:28.753115 4152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [term 1 LEADER]: Becoming Leader. State: Replica: 8e3e743bcec647e0bc9eeeef33adccf9, State: Running, Role: LEADER
I20260924 17:43:28.753268 4147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:28.753248 4152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44993 } }
I20260924 17:43:28.753691 4152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e3e743bcec647e0bc9eeeef33adccf9. Latest consensus state: current_term: 1 leader_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44993 } } }
I20260924 17:43:28.753932 4152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:28.753855 4153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e3e743bcec647e0bc9eeeef33adccf9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44993 } } }
I20260924 17:43:28.754333 4153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:28.754549 4161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:28.755162 4161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:28.755808 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:28.744120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.1.8.190:44993)
0924 17:43:28.744325 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:28.744334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:28.744367 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:43:28.744984 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:28.744994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:28.745066 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:28.745231 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:28.745944 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:28.747354 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:28.748184 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:28.748830 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:28.748917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:28.751615 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:28.751620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:28.751620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:28.751631 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:28.753800 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:28.754265 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:28.754267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:28.754268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:28.754380 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:28.754682 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:28.754683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:28.754683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:28.755016 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:28.755173 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:28.755286 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 17:43:28.755427 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260924 17:43:28.756873 4161 catalog_manager.cc:1347] Generated new cluster ID: 0741456e54004971835ed95d98835714
I20260924 17:43:28.756937 4161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:28.774919 4161 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:28.775545 4161 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:28.781020 4161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e3e743bcec647e0bc9eeeef33adccf9: Generated new TSK 0
I20260924 17:43:28.781256 4161 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:28.813822 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44993
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:28.905607 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:28.905853 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:28.905880 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:28.907119 4173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:28.907164 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:28.907192 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:28.908609 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:28.908828 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:28.909018 4173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:43:28.914376 4173 server_base.cc:1034] running on GCE node
W20260924 17:43:28.914304 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:28.914378 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:28.914304 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:28.914767 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:28.915026 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:28.916231 4173 hybrid_clock.cc:648] HybridClock initialized: now 1790271808916196 us; error 46 us; skew 500 ppm
Sep 24 17:43:28 dist-test-slave-r7hf krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:28.919078 4173 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:28.919593 4173 webserver.cc:460] Webserver started at http://127.1.8.129:44021/ using document root <none> and password file <none>
I20260924 17:43:28.919840 4173 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:28.919911 4173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:28.920065 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:28.921247 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "77ba57e78b3f4f3cb77eb0aab010e1be"
format_stamp: "Formatted at 2026-09-24 17:43:28 on dist-test-slave-r7hf"
server_key: "bdf1a1bb26933ee3f4f6755ced0cee0e"
server_key_iv: "66238db5132402626f27a7cd5e365c33"
server_key_version: "encryptionkey@0"
I20260924 17:43:28.921640 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "77ba57e78b3f4f3cb77eb0aab010e1be"
format_stamp: "Formatted at 2026-09-24 17:43:28 on dist-test-slave-r7hf"
server_key: "bdf1a1bb26933ee3f4f6755ced0cee0e"
server_key_iv: "66238db5132402626f27a7cd5e365c33"
server_key_version: "encryptionkey@0"
I20260924 17:43:28.923136 4173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:28.923934 4188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:28.924165 4173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:28.924283 4173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "77ba57e78b3f4f3cb77eb0aab010e1be"
format_stamp: "Formatted at 2026-09-24 17:43:28 on dist-test-slave-r7hf"
server_key: "bdf1a1bb26933ee3f4f6755ced0cee0e"
server_key_iv: "66238db5132402626f27a7cd5e365c33"
server_key_version: "encryptionkey@0"
I20260924 17:43:28.924363 4173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:28.958004 4173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:28.960351 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:28.960470 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:28.960779 4173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:28.961112 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:28.961145 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:28.961169 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:28.961220 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:28.967805 4173 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:46805
I20260924 17:43:28.967850 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:46805 every 8 connection(s)
I20260924 17:43:28.968205 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:43:28.970413 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4173
I20260924 17:43:28.970527 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:28.970700 1058 external_mini_cluster.cc:1428] Setting key 97db8b910cb914c9dedc5f76c726c424
Sep 24 17:43:28 dist-test-slave-r7hf krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:43:28.979971 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:28.968974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:33435 (local address 127.1.8.190:44993)
0924 17:43:28.969045 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:28.969047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:28.969292 (+ 245us) server_negotiation.cc:407] Connection header received
0924 17:43:28.969818 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:28.969819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:28.969828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:28.969882 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:28.971093 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:28.971814 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:28.973194 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:28.973441 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:28.973549 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:28.976365 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:28.976368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:28.976368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:28.976377 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:28.978139 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:28.978745 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:28.978747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:28.978748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:28.978838 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:28.979140 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:28.979142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:28.979142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:28.979268 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:28.979327 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:28.979733 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 17:43:28.979813 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:43:28.980108 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:28.968951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44993 (local address 127.1.8.129:33435)
0924 17:43:28.969162 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:28.969203 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:28.969722 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:28.969940 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:28.969943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:28.970695 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:28.970994 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:28.971001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:28.971947 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:28.971948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:28.973065 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:28.973068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:28.973560 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:28.973562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:28.973725 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:28.974359 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:28.974369 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:28.976209 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:28.978263 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:28.978273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:28.978276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:28.978639 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:28.978926 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:28.978927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:28.978927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:28.979044 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:28.979359 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:28.979365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:28.979513 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 17:43:28.979675 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 17:43:28.979896 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260924 17:43:28.980913 4302 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44993
I20260924 17:43:28.981022 4302 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:28.981204 4302 heartbeater.cc:511] Master 127.1.8.190:44993 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
I20260924 17:43:28.981866 4111 ts_manager.cc:194] Registered new tserver with Master: 77ba57e78b3f4f3cb77eb0aab010e1be (127.1.8.129:46805)
I20260924 17:43:28.982722 4111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:33435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:28.998584 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44993
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:29.092190 4307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:29.092428 4307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:29.092456 4307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:29.093964 4307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:29.094018 4307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:29.094053 4307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:29.095783 4307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:29.096081 4307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:29.096349 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:29.102082 4313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:29.102106 4312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:29.102113 4315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:29.102415 4307 server_base.cc:1034] running on GCE node
I20260924 17:43:29.102653 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:29.102942 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:29.104110 4307 hybrid_clock.cc:648] HybridClock initialized: now 1790271809104101 us; error 33 us; skew 500 ppm
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:29.106837 4307 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:29.107342 4307 webserver.cc:460] Webserver started at http://127.1.8.130:40677/ using document root <none> and password file <none>
I20260924 17:43:29.107602 4307 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:29.107679 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:29.107836 4307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:29.108958 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "672d4d9eeaee498ab925725074368541"
format_stamp: "Formatted at 2026-09-24 17:43:29 on dist-test-slave-r7hf"
server_key: "98fa6fcf51deccdc6b3ca0baa1284d37"
server_key_iv: "9eabf2f1b7c4177039afc31164d19461"
server_key_version: "encryptionkey@0"
I20260924 17:43:29.109344 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "672d4d9eeaee498ab925725074368541"
format_stamp: "Formatted at 2026-09-24 17:43:29 on dist-test-slave-r7hf"
server_key: "98fa6fcf51deccdc6b3ca0baa1284d37"
server_key_iv: "9eabf2f1b7c4177039afc31164d19461"
server_key_version: "encryptionkey@0"
I20260924 17:43:29.110826 4307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:29.111847 4322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:29.112143 4307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:29.112253 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "672d4d9eeaee498ab925725074368541"
format_stamp: "Formatted at 2026-09-24 17:43:29 on dist-test-slave-r7hf"
server_key: "98fa6fcf51deccdc6b3ca0baa1284d37"
server_key_iv: "9eabf2f1b7c4177039afc31164d19461"
server_key_version: "encryptionkey@0"
I20260924 17:43:29.112331 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:29.128926 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:29.131096 4307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:29.131248 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:29.131512 4307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:29.131852 4307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:29.131886 4307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:29.131928 4307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:29.131956 4307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:29.138315 4307 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39939
I20260924 17:43:29.138356 4435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39939 every 8 connection(s)
I20260924 17:43:29.138736 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:29.144397 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4307
I20260924 17:43:29.144528 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:29.144729 1058 external_mini_cluster.cc:1428] Setting key b2d045e57bf4e6f641168a908b02671d
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:29.149362 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:29.139527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:55213 (local address 127.1.8.190:44993)
0924 17:43:29.139622 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:29.139625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:29.139839 (+ 214us) server_negotiation.cc:407] Connection header received
0924 17:43:29.140401 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:29.140403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:29.140414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:29.140472 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:29.141499 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.141998 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.143110 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.143309 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.143401 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:29.145954 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:29.145955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:29.145956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:29.145969 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:29.147877 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:29.148314 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:29.148316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:29.148316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:29.148385 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:29.148658 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:29.148659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:29.148660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:29.148787 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:29.148866 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:29.149154 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 17:43:29.149241 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:43:29.149637 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:29.139512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44993 (local address 127.1.8.130:55213)
0924 17:43:29.139689 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:29.139745 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:29.140286 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:29.140580 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:29.140584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:29.141180 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:29.141382 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:29.141387 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.142101 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.142102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:29.143010 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:29.143012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.143414 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.143415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:29.143547 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:29.144124 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:29.144134 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:29.145852 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:29.147953 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:29.147962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:29.147965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:29.148219 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:29.148453 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:29.148454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:29.148454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:29.148578 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:29.148857 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:29.148864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:29.149018 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 17:43:29.149134 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 17:43:29.149361 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260924 17:43:29.150229 4436 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44993
I20260924 17:43:29.150347 4436 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:29.150556 4436 heartbeater.cc:511] Master 127.1.8.190:44993 requested a full tablet report, sending...
I20260924 17:43:29.150995 4111 ts_manager.cc:194] Registered new tserver with Master: 672d4d9eeaee498ab925725074368541 (127.1.8.130:39939)
I20260924 17:43:29.151389 4111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:55213
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:29.174610 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44993
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:29.259631 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:29.259863 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:29.259889 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:29.261188 4441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:29.261245 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:29.261276 4441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:29.263258 4441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:29.263567 4441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:29.263818 4441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:29.269044 4449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:29.269052 4446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:29.269068 4447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:29.269296 4441 server_base.cc:1034] running on GCE node
I20260924 17:43:29.269548 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:29.269802 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:29.270967 4441 hybrid_clock.cc:648] HybridClock initialized: now 1790271809270953 us; error 29 us; skew 500 ppm
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:29.273639 4441 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:29.274127 4441 webserver.cc:460] Webserver started at http://127.1.8.131:43669/ using document root <none> and password file <none>
I20260924 17:43:29.274340 4441 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:29.274384 4441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:29.274549 4441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:29.275628 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "d51e15cadce54101a971e8d3a39992b4"
format_stamp: "Formatted at 2026-09-24 17:43:29 on dist-test-slave-r7hf"
server_key: "c8d996ba8c37842e2fe45db18d4ba2c6"
server_key_iv: "02b5c7b55a7b50d15ce4684edd9a3874"
server_key_version: "encryptionkey@0"
I20260924 17:43:29.275941 4441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "d51e15cadce54101a971e8d3a39992b4"
format_stamp: "Formatted at 2026-09-24 17:43:29 on dist-test-slave-r7hf"
server_key: "c8d996ba8c37842e2fe45db18d4ba2c6"
server_key_iv: "02b5c7b55a7b50d15ce4684edd9a3874"
server_key_version: "encryptionkey@0"
I20260924 17:43:29.277299 4441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:29.278040 4456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:29.278255 4441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:29.278362 4441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "d51e15cadce54101a971e8d3a39992b4"
format_stamp: "Formatted at 2026-09-24 17:43:29 on dist-test-slave-r7hf"
server_key: "c8d996ba8c37842e2fe45db18d4ba2c6"
server_key_iv: "02b5c7b55a7b50d15ce4684edd9a3874"
server_key_version: "encryptionkey@0"
I20260924 17:43:29.278481 4441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:29.291923 4441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:29.294466 4441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:29.294613 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:29.294816 4441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:29.295156 4441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:29.295192 4441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:29.295248 4441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:29.295277 4441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:29.301127 4441 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:37605
I20260924 17:43:29.301191 4569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:37605 every 8 connection(s)
I20260924 17:43:29.301545 4441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271809, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:29.309952 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4441
I20260924 17:43:29.310077 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:29.310276 1058 external_mini_cluster.cc:1428] Setting key e2f3bc90a61dae0405ce779ba76188ec
I20260924 17:43:29.311970 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:29.302349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:48351 (local address 127.1.8.190:44993)
0924 17:43:29.302450 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:29.302452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:29.302649 (+ 197us) server_negotiation.cc:407] Connection header received
0924 17:43:29.303150 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:29.303151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:29.303163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:29.303223 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:29.304242 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.304747 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.305736 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.305929 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.305987 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:29.308292 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:29.308293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:29.308293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:29.308300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:29.309928 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:29.310618 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:29.310619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:29.310619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:29.310708 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:29.311085 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:29.311086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:29.311087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:29.311229 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:29.311319 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:29.311781 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 17:43:29.311876 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 17:43:29.312217 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:29.302320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44993 (local address 127.1.8.131:48351)
0924 17:43:29.302516 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:29.302555 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:29.303049 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:29.303306 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:29.303309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:29.303914 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:29.304116 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:29.304121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.304827 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.304827 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:29.305652 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:29.305655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.306005 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.306006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:29.306113 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:29.306701 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:29.306710 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:29.308197 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:29.310084 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:29.310093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:29.310096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:29.310483 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:29.310784 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:29.310785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:29.310785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:29.310996 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:29.311310 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:29.311317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:29.311483 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 17:43:29.311737 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 17:43:29.311934 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":123,"threads_started":1}
I20260924 17:43:29.312820 4570 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44993
I20260924 17:43:29.312929 4570 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:29.313082 4570 heartbeater.cc:511] Master 127.1.8.190:44993 requested a full tablet report, sending...
I20260924 17:43:29.313489 4111 ts_manager.cc:194] Registered new tserver with Master: d51e15cadce54101a971e8d3a39992b4 (127.1.8.131:37605)
I20260924 17:43:29.313902 4111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:48351
I20260924 17:43:29.322263 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271809, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271809, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:29.345377 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:29.338315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.1.8.190:44993)
0924 17:43:29.338447 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:29.338449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:29.338532 (+ 83us) server_negotiation.cc:407] Connection header received
0924 17:43:29.338586 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:29.338587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:29.338597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:29.338651 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:29.339281 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.339771 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.340510 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:29.340708 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:29.340758 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:29.343104 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:29.343106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:29.343106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:29.343114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:29.344361 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:29.344682 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:29.344683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:29.344683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:29.344732 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:29.344890 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:29.344891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:29.344891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:29.345037 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:29.345113 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:29.345217 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 17:43:29.345275 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260924 17:43:29.345667 4111 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49080
I20260924 17:43:29.346124 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4173
I20260924 17:43:29.350711 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4307
I20260924 17:43:29.355436 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4441
I20260924 17:43:29.360106 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4079
2026-09-24T17:43:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1924 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4582](info): set up 2 sockets
Sep 24 17:43:29 dist-test-slave-r7hf krb5kdc[4582](info): commencing operation
krb5kdc: starting...
W20260924 17:43:30.410472 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:30 dist-test-slave-r7hf krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:30.537786 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:43467
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:43467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:30.623349 4595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:30.623580 4595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:30.623613 4595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:30.625311 4595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:30.625384 4595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:30.625411 4595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:30.625437 4595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:30.625460 4595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:30.628443 4595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:43467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:43467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:30.628813 4595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:30.629123 4595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:30.635175 4601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:30.635262 4600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:30.635172 4603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:30.635378 4595 server_base.cc:1034] running on GCE node
I20260924 17:43:30.635635 4595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:30.635962 4595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:30.637136 4595 hybrid_clock.cc:648] HybridClock initialized: now 1790271810637117 us; error 54 us; skew 500 ppm
Sep 24 17:43:30 dist-test-slave-r7hf krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:30.639904 4595 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:30.640473 4595 webserver.cc:460] Webserver started at http://127.1.8.190:35513/ using document root <none> and password file <none>
I20260924 17:43:30.640744 4595 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:30.640818 4595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:30.640975 4595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:30.642081 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "9f1f085706ee4b0b9c96d675ee4bad3a"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "164c775b3b4a038f773b9b27d635e3a1"
server_key_iv: "d31cf9c64d43d48c3cdb73a022a5bd17"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.642549 4595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "9f1f085706ee4b0b9c96d675ee4bad3a"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "164c775b3b4a038f773b9b27d635e3a1"
server_key_iv: "d31cf9c64d43d48c3cdb73a022a5bd17"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.643992 4595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:30.644905 4610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:30.645159 4595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:43:30.645274 4595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "9f1f085706ee4b0b9c96d675ee4bad3a"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "164c775b3b4a038f773b9b27d635e3a1"
server_key_iv: "d31cf9c64d43d48c3cdb73a022a5bd17"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.645365 4595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:30.653861 4595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:30.656381 4595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:30.656566 4595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:30.660573 4595 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:43467
I20260924 17:43:30.660883 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:43467 every 8 connection(s)
I20260924 17:43:30.660980 4595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:30.661769 4663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:30.662890 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4595
I20260924 17:43:30.663004 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:30.663188 1058 external_mini_cluster.cc:1428] Setting key 3c665d71116029a55d11b10dfc1fc98b
I20260924 17:43:30.664729 4663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a: Bootstrap starting.
I20260924 17:43:30.665613 4663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:30.665961 4663 log.cc:824] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:30.666723 4663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a: No bootstrap required, opened a new log
I20260924 17:43:30.669236 4663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43467 } }
I20260924 17:43:30.669420 4663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:30.669466 4663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1f085706ee4b0b9c96d675ee4bad3a, State: Initialized, Role: FOLLOWER
I20260924 17:43:30.669610 4663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43467 } }
I20260924 17:43:30.669692 4663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:30.669739 4663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:30.669790 4663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:30.670485 4663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43467 } }
I20260924 17:43:30.670626 4663 leader_election.cc:302] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f1f085706ee4b0b9c96d675ee4bad3a; no voters:
I20260924 17:43:30.670849 4663 leader_election.cc:288] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:30.670877 4668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:30.670995 4668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [term 1 LEADER]: Becoming Leader. State: Replica: 9f1f085706ee4b0b9c96d675ee4bad3a, State: Running, Role: LEADER
I20260924 17:43:30.671080 4668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43467 } }
I20260924 17:43:30.671170 4663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 17:43:30 dist-test-slave-r7hf krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:30.671416 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43467 } } }
I20260924 17:43:30.671491 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:30.671613 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f1f085706ee4b0b9c96d675ee4bad3a. Latest consensus state: current_term: 1 leader_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f1f085706ee4b0b9c96d675ee4bad3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43467 } } }
I20260924 17:43:30.671684 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a [sys.catalog]: This master's current role is: LEADER
W20260924 17:43:30.673005 4683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:30.673084 4683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:30.673296 4672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:30.673486 4672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:30.674988 4672 catalog_manager.cc:1347] Generated new cluster ID: 7a70ddd37f8d4b23870e4eb9ae559b9e
I20260924 17:43:30.675047 4672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:30.676985 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:30.663858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.1.8.190:43467)
0924 17:43:30.664054 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:30.664071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:30.664099 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:43:30.664593 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:30.664601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:30.664656 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:30.664794 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:30.665724 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:30.667042 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:30.668104 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:30.668578 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:30.668683 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:30.671906 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:30.671911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:30.671912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:30.671927 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:30.675197 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:30.675669 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:30.675673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:30.675673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:30.675785 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:30.676027 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:30.676027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:30.676028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:30.676326 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:30.676461 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:30.676644 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:43:30.676783 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260924 17:43:30.690514 4672 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:30.691177 4672 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:30.695182 4672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f1f085706ee4b0b9c96d675ee4bad3a: Generated new TSK 0
I20260924 17:43:30.695447 4672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:30.726922 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43467
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:30.812925 4689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:30.813151 4689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:30.813179 4689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:30.814487 4689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:30.814538 4689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:30.814569 4689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:30.816184 4689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:30.816450 4689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:30.816735 4689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:30.822024 4694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:30.822135 4695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:30.822166 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:30.822378 4689 server_base.cc:1034] running on GCE node
I20260924 17:43:30.822651 4689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:30.822878 4689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:30.824074 4689 hybrid_clock.cc:648] HybridClock initialized: now 1790271810824065 us; error 36 us; skew 500 ppm
Sep 24 17:43:30 dist-test-slave-r7hf krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:30.826906 4689 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:30.827358 4689 webserver.cc:460] Webserver started at http://127.1.8.129:32783/ using document root <none> and password file <none>
I20260924 17:43:30.827661 4689 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:30.827739 4689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:30.827890 4689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:30.828969 4689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "1a2ece03fbf648b59b5119e30b248d00"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "3f94c5477738a2950eff2fab911498cb"
server_key_iv: "5600893536cc5952a7b524779475a9dc"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.829382 4689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "1a2ece03fbf648b59b5119e30b248d00"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "3f94c5477738a2950eff2fab911498cb"
server_key_iv: "5600893536cc5952a7b524779475a9dc"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.830801 4689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:30.831527 4704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:30.831722 4689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:30.831842 4689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "1a2ece03fbf648b59b5119e30b248d00"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "3f94c5477738a2950eff2fab911498cb"
server_key_iv: "5600893536cc5952a7b524779475a9dc"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.831918 4689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:30.846169 4689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:30.848318 4689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:30.848508 4689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:30.848799 4689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:30.849131 4689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:30.849188 4689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:30.849234 4689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:30.849260 4689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:30.855319 4689 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:38911
I20260924 17:43:30.855360 4817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:38911 every 8 connection(s)
I20260924 17:43:30.855719 4689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:30 dist-test-slave-r7hf krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:30.862514 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4689
I20260924 17:43:30.862627 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:30.862818 1058 external_mini_cluster.cc:1428] Setting key 15beef6d5d1288bf24d50581bb3eb2e1
I20260924 17:43:30.865971 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:30.856476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56775 (local address 127.1.8.190:43467)
0924 17:43:30.856604 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:30.856606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:30.856733 (+ 127us) server_negotiation.cc:407] Connection header received
0924 17:43:30.857771 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:30.857772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:30.857785 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:30.857853 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:30.858618 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:30.859163 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:30.860204 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:30.860409 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:30.860464 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:30.863001 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:30.863004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:30.863004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:30.863011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:30.864527 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:30.864973 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:30.864974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:30.864974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:30.865033 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:30.865313 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:30.865314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:30.865314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:30.865431 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:30.865515 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:30.865790 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 17:43:30.865876 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 17:43:30.866196 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:30.856429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:43467 (local address 127.1.8.129:56775)
0924 17:43:30.856606 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:30.856652 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:30.857615 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:30.857942 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:30.857946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:30.858235 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:30.858519 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:30.858524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:30.859254 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:30.859255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:30.860119 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:30.860121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:30.860482 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:30.860483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:30.860611 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:30.861118 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:30.861127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:30.862887 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:30.864633 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:30.864642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:30.864644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:30.864885 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:30.865112 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:30.865113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:30.865114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:30.865236 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:30.865498 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:30.865504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:30.865656 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 17:43:30.865770 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 17:43:30.865938 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260924 17:43:30.866803 4818 heartbeater.cc:347] Connected to a master server at 127.1.8.190:43467
I20260924 17:43:30.866905 4818 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:30.867116 4818 heartbeater.cc:511] Master 127.1.8.190:43467 requested a full tablet report, sending...
I20260924 17:43:30.867722 4627 ts_manager.cc:194] Registered new tserver with Master: 1a2ece03fbf648b59b5119e30b248d00 (127.1.8.129:38911)
I20260924 17:43:30.868655 4627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:56775
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:30.890724 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43467
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:30.974737 4823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:30.975036 4823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:30.975086 4823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:30.976358 4823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:30.976442 4823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:30.976500 4823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:30.978117 4823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:30.978446 4823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:30.978717 4823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:30.984097 4828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:30.984052 4829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:30.984052 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:30.984318 4823 server_base.cc:1034] running on GCE node
I20260924 17:43:30.984558 4823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:30.984827 4823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:30.986008 4823 hybrid_clock.cc:648] HybridClock initialized: now 1790271810985989 us; error 46 us; skew 500 ppm
Sep 24 17:43:30 dist-test-slave-r7hf krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:30.988790 4823 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:30.989414 4823 webserver.cc:460] Webserver started at http://127.1.8.130:42027/ using document root <none> and password file <none>
I20260924 17:43:30.989675 4823 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:30.989748 4823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:30.989905 4823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:30.991246 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "b7c16bd2963d4f66b7d3f06c0fda33b3"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "6f64127f8afdf39fb8c211267e4320f5"
server_key_iv: "6423dfc68b374282eacd353b59b00392"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.991653 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "b7c16bd2963d4f66b7d3f06c0fda33b3"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "6f64127f8afdf39fb8c211267e4320f5"
server_key_iv: "6423dfc68b374282eacd353b59b00392"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.993099 4823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:43:30.993925 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:30.994131 4823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:30.994246 4823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "b7c16bd2963d4f66b7d3f06c0fda33b3"
format_stamp: "Formatted at 2026-09-24 17:43:30 on dist-test-slave-r7hf"
server_key: "6f64127f8afdf39fb8c211267e4320f5"
server_key_iv: "6423dfc68b374282eacd353b59b00392"
server_key_version: "encryptionkey@0"
I20260924 17:43:30.994328 4823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:31.011261 4823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:31.013989 4823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:31.014185 4823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:31.014503 4823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:31.014859 4823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:31.014931 4823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:31.014969 4823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:31.015000 4823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:31.021435 4823 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39211
I20260924 17:43:31.021472 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39211 every 8 connection(s)
I20260924 17:43:31.021842 4823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:31.026028 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4823
I20260924 17:43:31.026126 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:31.026325 1058 external_mini_cluster.cc:1428] Setting key 454e3855a0d7d9b592e83b0c54690adf
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271810, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:31.032367 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:31.022709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33151 (local address 127.1.8.190:43467)
0924 17:43:31.022776 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:31.022778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:31.022963 (+ 185us) server_negotiation.cc:407] Connection header received
0924 17:43:31.024004 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:31.024005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:31.024015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:31.024071 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:31.024714 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.025251 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.026463 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.026708 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.026809 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:31.029147 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:31.029148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:31.029149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:31.029159 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:31.030770 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.031316 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.031317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.031318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.031379 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.031667 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.031667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.031668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.031796 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:31.031867 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:31.032176 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 17:43:31.032271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:43:31.032589 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:31.022690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:43467 (local address 127.1.8.130:33151)
0924 17:43:31.022856 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:31.022894 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:31.023877 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:31.024136 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:31.024139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:31.024413 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:31.024614 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:31.024618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.025345 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.025346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:31.026328 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:31.026331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.026815 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.026816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:31.026926 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:31.027570 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:31.027600 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:31.029032 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:31.030875 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:31.030883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:31.030885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:31.031222 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:31.031449 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:31.031450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:31.031450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:31.031588 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:31.031870 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:31.031877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:31.032032 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 17:43:31.032142 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 17:43:31.032316 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260924 17:43:31.033066 4952 heartbeater.cc:347] Connected to a master server at 127.1.8.190:43467
I20260924 17:43:31.033169 4952 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:31.033381 4952 heartbeater.cc:511] Master 127.1.8.190:43467 requested a full tablet report, sending...
I20260924 17:43:31.033782 4627 ts_manager.cc:194] Registered new tserver with Master: b7c16bd2963d4f66b7d3f06c0fda33b3 (127.1.8.130:39211)
I20260924 17:43:31.034176 4627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:33151
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:31.054718 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43467
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:31.138247 4957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:31.138573 4957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:31.138619 4957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:31.139876 4957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:31.139961 4957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:31.140018 4957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:31.141776 4957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:31.142095 4957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:31.142344 4957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:31.147912 4962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:31.147912 4963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:31.147913 4965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:31.148252 4957 server_base.cc:1034] running on GCE node
I20260924 17:43:31.148481 4957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:31.148782 4957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:31.149950 4957 hybrid_clock.cc:648] HybridClock initialized: now 1790271811149929 us; error 34 us; skew 500 ppm
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271811, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:31.152712 4957 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:31.153215 4957 webserver.cc:460] Webserver started at http://127.1.8.131:46093/ using document root <none> and password file <none>
I20260924 17:43:31.153457 4957 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:31.153528 4957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:31.153688 4957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:31.154969 4957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "b49e935ff6274f00831b6ae38aae2c9d"
format_stamp: "Formatted at 2026-09-24 17:43:31 on dist-test-slave-r7hf"
server_key: "162db9552661ad761bc988f494460858"
server_key_iv: "335e1b2bdd34445926cf6db8712c4b80"
server_key_version: "encryptionkey@0"
I20260924 17:43:31.155344 4957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "b49e935ff6274f00831b6ae38aae2c9d"
format_stamp: "Formatted at 2026-09-24 17:43:31 on dist-test-slave-r7hf"
server_key: "162db9552661ad761bc988f494460858"
server_key_iv: "335e1b2bdd34445926cf6db8712c4b80"
server_key_version: "encryptionkey@0"
I20260924 17:43:31.156810 4957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:31.157729 4972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:31.157948 4957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:31.158057 4957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "b49e935ff6274f00831b6ae38aae2c9d"
format_stamp: "Formatted at 2026-09-24 17:43:31 on dist-test-slave-r7hf"
server_key: "162db9552661ad761bc988f494460858"
server_key_iv: "335e1b2bdd34445926cf6db8712c4b80"
server_key_version: "encryptionkey@0"
I20260924 17:43:31.158138 4957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:31.174743 4957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:31.177474 4957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:31.177649 4957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:31.177929 4957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:31.178260 4957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:31.178315 4957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:31.178339 4957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:31.178386 4957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:31.184535 4957 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:41591
I20260924 17:43:31.184615 5085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:41591 every 8 connection(s)
I20260924 17:43:31.184990 4957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:43:31.189991 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 4957
I20260924 17:43:31.190121 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:31.190344 1058 external_mini_cluster.cc:1428] Setting key 3c07937f0c4b875c31e3a2debe6c2272
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271811, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:31.196234 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:31.185852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:41111 (local address 127.1.8.190:43467)
0924 17:43:31.185921 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:31.185925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:31.186202 (+ 277us) server_negotiation.cc:407] Connection header received
0924 17:43:31.187273 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:31.187275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:31.187286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:31.187346 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:31.188052 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.188609 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.189661 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.189872 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.190024 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:31.192844 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:31.192846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:31.192847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:31.192855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:31.194588 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.195173 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.195174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.195175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.195239 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.195543 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.195544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.195544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.195701 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:31.195777 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:31.196079 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 17:43:31.196159 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:43:31.196403 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:31.185844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:43467 (local address 127.1.8.131:41111)
0924 17:43:31.186026 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:31.186067 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:31.187143 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:31.187430 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:31.187433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:31.187719 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:31.187929 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:31.187933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.188704 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.188705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:31.189538 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:31.189540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.190039 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.190040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:31.190236 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:31.190992 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:31.191002 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:31.192716 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:31.194699 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:31.194715 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:31.194718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:31.195077 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:31.195326 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:31.195327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:31.195327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:31.195461 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:31.195776 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:31.195782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:31.195921 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 17:43:31.196021 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 17:43:31.196179 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260924 17:43:31.196971 5086 heartbeater.cc:347] Connected to a master server at 127.1.8.190:43467
I20260924 17:43:31.197088 5086 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:31.197317 5086 heartbeater.cc:511] Master 127.1.8.190:43467 requested a full tablet report, sending...
I20260924 17:43:31.197764 4627 ts_manager.cc:194] Registered new tserver with Master: b49e935ff6274f00831b6ae38aae2c9d (127.1.8.131:41591)
I20260924 17:43:31.198223 4627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:41111
I20260924 17:43:31.202276 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[4582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:31.226083 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:31.217776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.1.8.129:38911)
0924 17:43:31.217958 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:31.217970 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:31.217982 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:43:31.218010 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:31.218012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:31.218026 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:31.218095 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:31.218813 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.219545 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.220324 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.220686 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.220743 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:31.223183 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:31.223194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:31.223194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:31.223202 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:31.224993 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.225339 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.225342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.225343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.225411 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.225591 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.225591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.225592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.225759 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:31.225878 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:31.225933 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:43:31.226000 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[4582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:31.234885 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:31.227530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.1.8.190:43467)
0924 17:43:31.227603 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:31.227604 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:31.227624 (+ 20us) server_negotiation.cc:407] Connection header received
0924 17:43:31.227688 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:31.227689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:31.227697 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:31.227745 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:31.228256 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.228823 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.229477 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:31.229722 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:31.229784 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:31.232163 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:31.232164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:31.232165 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:31.232173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:31.233642 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.233965 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.233966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.233966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.234016 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:31.234245 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:31.234245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:31.234246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:31.234456 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:31.234536 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:31.234748 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 17:43:31.234816 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 17:43:31.235235 4627 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57304
I20260924 17:43:31.235724 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4689
I20260924 17:43:31.241508 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4823
I20260924 17:43:31.246152 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4957
I20260924 17:43:31.250792 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 4595
2026-09-24T17:43:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1890 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[5098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[5098](info): set up 2 sockets
Sep 24 17:43:31 dist-test-slave-r7hf krb5kdc[5098](info): commencing operation
krb5kdc: starting...
W20260924 17:43:32.335021 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:32.462059 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:36119
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:36119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:32.553933 5111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:32.554256 5111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:32.554304 5111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:32.555711 5111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:32.555807 5111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:32.555837 5111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:32.555872 5111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:32.555927 5111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:32.557749 5111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:36119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:36119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:32.558084 5111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:32.558342 5111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:32.563709 5117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:32.563773 5119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:32.563839 5111 server_base.cc:1034] running on GCE node
W20260924 17:43:32.563803 5116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:32.564275 5111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:32.564615 5111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:32.565795 5111 hybrid_clock.cc:648] HybridClock initialized: now 1790271812565771 us; error 48 us; skew 500 ppm
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:32.574860 5111 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:32.575258 5111 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:32.575331 5111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:32.575488 5111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:32.576653 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "a6275462e23d4ed5959f054521d6fb8c"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "b4615eb13b0c167148dc1fa112ea1cbd"
server_key_iv: "71bd703d96c5b62f7d04043520ce2ca2"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.577041 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "a6275462e23d4ed5959f054521d6fb8c"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "b4615eb13b0c167148dc1fa112ea1cbd"
server_key_iv: "71bd703d96c5b62f7d04043520ce2ca2"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.578444 5111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 17:43:32.579286 5125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.579473 5111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:32.579588 5111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "a6275462e23d4ed5959f054521d6fb8c"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "b4615eb13b0c167148dc1fa112ea1cbd"
server_key_iv: "71bd703d96c5b62f7d04043520ce2ca2"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.579701 5111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:32.603446 5111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:32.606045 5111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:32.606230 5111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:32.610318 5111 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:36119
I20260924 17:43:32.610344 5177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:36119 every 8 connection(s)
I20260924 17:43:32.610703 5111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:32.611707 5178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:32.614130 5178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c: Bootstrap starting.
I20260924 17:43:32.614787 5178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:32.615118 5178 log.cc:824] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:32.615821 5178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c: No bootstrap required, opened a new log
I20260924 17:43:32.617472 5178 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6275462e23d4ed5959f054521d6fb8c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36119 } }
I20260924 17:43:32.617630 5178 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:32.617652 5178 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6275462e23d4ed5959f054521d6fb8c, State: Initialized, Role: FOLLOWER
I20260924 17:43:32.617812 5178 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6275462e23d4ed5959f054521d6fb8c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36119 } }
I20260924 17:43:32.617887 5178 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:32.617920 5178 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:32.617897 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5111
I20260924 17:43:32.617975 5178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:32.618022 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:32.618208 1058 external_mini_cluster.cc:1428] Setting key 9e4b749b11263c5b62f6358b38c03697
I20260924 17:43:32.619171 5178 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6275462e23d4ed5959f054521d6fb8c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36119 } }
I20260924 17:43:32.619271 5178 leader_election.cc:302] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6275462e23d4ed5959f054521d6fb8c; no voters:
I20260924 17:43:32.619457 5178 leader_election.cc:288] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:32.619645 5184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:32.619756 5184 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [term 1 LEADER]: Becoming Leader. State: Replica: a6275462e23d4ed5959f054521d6fb8c, State: Running, Role: LEADER
I20260924 17:43:32.619760 5178 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:32.619921 5184 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6275462e23d4ed5959f054521d6fb8c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36119 } }
I20260924 17:43:32.620323 5183 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6275462e23d4ed5959f054521d6fb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6275462e23d4ed5959f054521d6fb8c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36119 } } }
I20260924 17:43:32.620456 5183 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:32.620357 5185 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6275462e23d4ed5959f054521d6fb8c. Latest consensus state: current_term: 1 leader_uuid: "a6275462e23d4ed5959f054521d6fb8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6275462e23d4ed5959f054521d6fb8c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36119 } } }
I20260924 17:43:32.620656 5185 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:32.620843 5192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:32.621431 5192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:32.623250 5192 catalog_manager.cc:1347] Generated new cluster ID: ca9029405c5a4ab7a0df376cb9f74850
I20260924 17:43:32.623301 5192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:32.631542 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:32.618817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.1.8.190:36119)
0924 17:43:32.619044 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:32.619053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:32.619081 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:43:32.619187 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:32.619194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:32.619266 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:32.619433 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:32.620242 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.621900 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.623011 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.623640 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.623744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:32.626744 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:32.626750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:32.626750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:32.626765 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:32.629600 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:32.630078 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:32.630082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:32.630082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:32.630237 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:32.630477 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:32.630478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:32.630478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:32.630874 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:32.631049 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:32.631137 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 17:43:32.631291 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":167,"thread_start_us":107,"threads_started":1}
I20260924 17:43:32.639169 5192 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:32.639955 5192 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:32.643231 5192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c: Generated new TSK 0
I20260924 17:43:32.643446 5192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:32.677403 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36119
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:32.763638 5204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:32.763885 5204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:32.763914 5204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:32.765229 5204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:32.765288 5204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:32.765321 5204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:32.767087 5204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:32.767321 5204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:32.767544 5204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:32.773150 5210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:32.773283 5212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:32.773150 5209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:32.773663 5204 server_base.cc:1034] running on GCE node
I20260924 17:43:32.773928 5204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:32.774248 5204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:32.775405 5204 hybrid_clock.cc:648] HybridClock initialized: now 1790271812775381 us; error 40 us; skew 500 ppm
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:32.778131 5204 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:32.778625 5204 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:32.778680 5204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:32.778857 5204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:32.779948 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "670269c1d35c471294c06768ebc0d5bd"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "e04d3767a906c9b708fec333a875dd05"
server_key_iv: "86a10eba808a328fad3cae27b64a84f1"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.780323 5204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "670269c1d35c471294c06768ebc0d5bd"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "e04d3767a906c9b708fec333a875dd05"
server_key_iv: "86a10eba808a328fad3cae27b64a84f1"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.781628 5204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:32.782475 5218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.782761 5204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:32.782846 5204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "670269c1d35c471294c06768ebc0d5bd"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "e04d3767a906c9b708fec333a875dd05"
server_key_iv: "86a10eba808a328fad3cae27b64a84f1"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.782903 5204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:32.796636 5204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:32.798949 5204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:32.799089 5204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:32.799299 5204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:32.799724 5204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:32.799770 5204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.799794 5204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:32.799847 5204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.806044 5204 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42173
I20260924 17:43:32.806123 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42173 every 8 connection(s)
I20260924 17:43:32.806478 5204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:32.812839 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5204
I20260924 17:43:32.812981 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:32.813179 1058 external_mini_cluster.cc:1428] Setting key ca671d4d832ce39d22d4e919825ff72f
I20260924 17:43:32.817083 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:32.807266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:53829 (local address 127.1.8.190:36119)
0924 17:43:32.807315 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:32.807318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:32.807532 (+ 214us) server_negotiation.cc:407] Connection header received
0924 17:43:32.808023 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:32.808025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:32.808034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:32.808101 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:32.809058 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.809565 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.810646 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.810852 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.810919 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:32.813372 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:32.813374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:32.813374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:32.813386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:32.815368 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:32.815938 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:32.815939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:32.815940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:32.816001 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:32.816320 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:32.816321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:32.816321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:32.816430 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:32.816558 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:32.816851 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 17:43:32.816946 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:43:32.817116 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:32.807216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:36119 (local address 127.1.8.129:53829)
0924 17:43:32.807392 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:32.807437 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:32.807937 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:32.808166 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:32.808170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:32.808446 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:32.808944 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:32.808949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.809655 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.809655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:32.810504 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:32.810506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.810936 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.810937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:32.811047 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:32.811663 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:32.811672 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:32.813229 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:32.815469 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:32.815480 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:32.815483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:32.815852 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:32.816072 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:32.816074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:32.816074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:32.816254 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:32.816493 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:32.816500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:32.816673 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 17:43:32.816804 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 17:43:32.816942 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260924 17:43:32.817813 5332 heartbeater.cc:347] Connected to a master server at 127.1.8.190:36119
I20260924 17:43:32.817935 5332 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:32.818135 5332 heartbeater.cc:511] Master 127.1.8.190:36119 requested a full tablet report, sending...
I20260924 17:43:32.818778 5142 ts_manager.cc:194] Registered new tserver with Master: 670269c1d35c471294c06768ebc0d5bd (127.1.8.129:42173)
I20260924 17:43:32.819416 5142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:53829
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:32.841696 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36119
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:32.926352 5337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:32.926651 5337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:32.926693 5337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:32.927917 5337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:32.927999 5337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:32.928047 5337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:32.929651 5337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:32.929949 5337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:32.930192 5337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:32.935451 5345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:32.935451 5343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:32.935499 5342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:32.935846 5337 server_base.cc:1034] running on GCE node
I20260924 17:43:32.936075 5337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:32.936332 5337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:32.937476 5337 hybrid_clock.cc:648] HybridClock initialized: now 1790271812937467 us; error 31 us; skew 500 ppm
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:32.940225 5337 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:32.940596 5337 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:32.940670 5337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:32.940831 5337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:32.941912 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "c513b942352c4cca9810c07a3877427d"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "a806c7d066ef8519945e1396b4ee7776"
server_key_iv: "eafeef5999e7240ef01663bb7fbc08db"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.942301 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "c513b942352c4cca9810c07a3877427d"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "a806c7d066ef8519945e1396b4ee7776"
server_key_iv: "eafeef5999e7240ef01663bb7fbc08db"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.943701 5337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:32.944535 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.944744 5337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:32.944854 5337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "c513b942352c4cca9810c07a3877427d"
format_stamp: "Formatted at 2026-09-24 17:43:32 on dist-test-slave-r7hf"
server_key: "a806c7d066ef8519945e1396b4ee7776"
server_key_iv: "eafeef5999e7240ef01663bb7fbc08db"
server_key_version: "encryptionkey@0"
I20260924 17:43:32.944934 5337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:32.965342 5337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:32.967909 5337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:32.968087 5337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:32.968366 5337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:32.968707 5337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:32.968761 5337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.968797 5337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:32.968837 5337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:32.975091 5337 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:40171
I20260924 17:43:32.975169 5464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:40171 every 8 connection(s)
I20260924 17:43:32.975476 5337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:32.977181 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5337
I20260924 17:43:32.977303 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:32.977473 1058 external_mini_cluster.cc:1428] Setting key 822cedfa4cc5af33be7439bc9ec45d5c
Sep 24 17:43:32 dist-test-slave-r7hf krb5kdc[5098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:43:32.986109 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:32.976307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:40555 (local address 127.1.8.190:36119)
0924 17:43:32.976376 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:32.976379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:32.976645 (+ 266us) server_negotiation.cc:407] Connection header received
0924 17:43:32.977138 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:32.977140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:32.977152 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:32.977197 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:32.978566 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.979072 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.980118 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.980335 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.980426 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:32.983068 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:32.983070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:32.983070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:32.983078 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:32.984562 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:32.985089 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:32.985090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:32.985090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:32.985143 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:32.985441 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:32.985442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:32.985442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:32.985547 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:32.985598 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:32.985966 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 17:43:32.986040 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:32.986454 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:32.976304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:36119 (local address 127.1.8.130:40555)
0924 17:43:32.976502 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:32.976542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:32.977023 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:32.977283 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:32.977295 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:32.977681 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:32.978458 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:32.978474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.979183 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.979184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:32.980011 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:32.980013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:32.980435 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:32.980437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:32.980576 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:32.981148 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:32.981157 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:32.982952 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:32.984653 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:32.984660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:32.984662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:32.984986 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:32.985214 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:32.985216 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:32.985216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:32.985373 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:32.985613 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:32.985619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:32.985805 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 17:43:32.985944 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 17:43:32.986145 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
I20260924 17:43:32.987207 5465 heartbeater.cc:347] Connected to a master server at 127.1.8.190:36119
I20260924 17:43:32.987362 5465 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:32.987543 5465 heartbeater.cc:511] Master 127.1.8.190:36119 requested a full tablet report, sending...
I20260924 17:43:32.988010 5142 ts_manager.cc:194] Registered new tserver with Master: c513b942352c4cca9810c07a3877427d (127.1.8.130:40171)
I20260924 17:43:32.988421 5142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:40555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:33.006469 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36119
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:33.091254 5470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:33.091478 5470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:33.091506 5470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:33.092689 5470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:33.092728 5470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:33.092754 5470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:33.094254 5470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:33.094544 5470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:33.094744 5470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:33.100148 5478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:33.100149 5470 server_base.cc:1034] running on GCE node
W20260924 17:43:33.100144 5475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.100158 5476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:33.100636 5470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:33.100852 5470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:33.102041 5470 hybrid_clock.cc:648] HybridClock initialized: now 1790271813102034 us; error 36 us; skew 500 ppm
Sep 24 17:43:33 dist-test-slave-r7hf krb5kdc[5098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271813, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:33.104809 5470 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:33.105144 5470 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:33.105190 5470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:33.105285 5470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:33.106346 5470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "659e2af60e244076a0d7e8b325cdd150"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "4222b3474c5a8abcd37d179f7c31b9b8"
server_key_iv: "f5d792bca4e4e5b29801442a71272bfc"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.106786 5470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "659e2af60e244076a0d7e8b325cdd150"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "4222b3474c5a8abcd37d179f7c31b9b8"
server_key_iv: "f5d792bca4e4e5b29801442a71272bfc"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.108155 5470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:33.108999 5484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.109189 5470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:33.109299 5470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "659e2af60e244076a0d7e8b325cdd150"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "4222b3474c5a8abcd37d179f7c31b9b8"
server_key_iv: "f5d792bca4e4e5b29801442a71272bfc"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.109377 5470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:33.122562 5470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:33.124926 5470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:33.125145 5470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:33.125432 5470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:33.125788 5470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:33.125826 5470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.125876 5470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:33.125906 5470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.132380 5470 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:36277
I20260924 17:43:33.132452 5597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:36277 every 8 connection(s)
I20260924 17:43:33.132783 5470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:33 dist-test-slave-r7hf krb5kdc[5098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271813, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:33.141741 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5470
I20260924 17:43:33.141902 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:33.142161 1058 external_mini_cluster.cc:1428] Setting key 6808996d6670a096f9573db5561b9392
I20260924 17:43:33.143255 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.133604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39501 (local address 127.1.8.190:36119)
0924 17:43:33.133683 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.133686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.133844 (+ 158us) server_negotiation.cc:407] Connection header received
0924 17:43:33.134354 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.134355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.134365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.134461 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.135513 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.136024 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.137122 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.137303 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.137359 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.139860 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.139862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.139862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:33.139871 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.141395 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:33.141856 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:33.141857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:33.141858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:33.141947 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:33.142322 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:33.142323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:33.142323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:33.142506 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.142582 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.143069 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.143162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:43:33.143555 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.133535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:36119 (local address 127.1.8.131:39501)
0924 17:43:33.133712 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.133752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.134263 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.134541 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.134545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.134834 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.135392 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.135397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.136123 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.136124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.137040 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.137042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.137387 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.137388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.137495 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.138076 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:33.138085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.139748 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.141498 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:33.141507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:33.141510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:33.141774 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:33.142033 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:33.142034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:33.142035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:33.142200 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:33.142641 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.142647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.142890 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.143038 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.143219 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260924 17:43:33.144013 5598 heartbeater.cc:347] Connected to a master server at 127.1.8.190:36119
I20260924 17:43:33.144114 5598 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:33.144244 5598 heartbeater.cc:511] Master 127.1.8.190:36119 requested a full tablet report, sending...
I20260924 17:43:33.144614 5142 ts_manager.cc:194] Registered new tserver with Master: 659e2af60e244076a0d7e8b325cdd150 (127.1.8.131:36277)
I20260924 17:43:33.144953 5142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:39501
I20260924 17:43:33.154294 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:43:33.162492 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.155752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.1.8.190:36119)
0924 17:43:33.155818 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.155820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.155834 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:43:33.155897 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.155898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.155907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.155969 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.156588 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.157106 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.157944 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.158142 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.158201 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.159604 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.159606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.159606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:33.159613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.161224 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:33.161609 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:33.161610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:33.161610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:33.161667 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:33.161850 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:33.161850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:33.161851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:33.162016 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.162096 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.162254 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.162328 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:33.164613 5142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:33.166460 5142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:33.173617 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.169207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:40171 (local address 127.0.0.1:43240)
0924 17:43:33.169517 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.169558 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.169629 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.169937 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.169940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.169942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:33.170039 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.170042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.171141 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.171143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.172389 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.172391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.173285 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.173286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.173365 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.173380 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.173458 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.173517 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260924 17:43:33.173863 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.169207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:36277 (local address 127.0.0.1:45080)
0924 17:43:33.169511 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.169558 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.169629 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.170249 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.170250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.170252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:33.170307 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.170309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.171678 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.171679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.172661 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.172663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.173630 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.173631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.173704 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.173709 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.173743 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.173795 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":54,"threads_started":1}
I20260924 17:43:33.173650 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.169570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.1.8.130:40171)
0924 17:43:33.169733 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.169737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.169747 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:33.169790 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.169792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.169811 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.169930 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:33.170129 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.171034 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.172466 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.173189 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.173268 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.173328 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.173497 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.173543 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260924 17:43:33.174197 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.169373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.1.8.131:36277)
0924 17:43:33.170076 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.170080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.170090 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:33.170130 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.170132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.170147 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.170382 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:33.170508 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.171566 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.172747 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.173532 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.173956 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.174033 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.174062 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.174115 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":89,"threads_started":1}
I20260924 17:43:33.174592 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.169409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42173 (local address 127.0.0.1:44534)
0924 17:43:33.169727 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.169737 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.169781 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.170025 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.170026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.170029 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:33.170163 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.170166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.172732 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.172733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.173489 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.173491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.174287 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.174288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.174361 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.174366 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.174485 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.174527 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":66,"threads_started":1}
I20260924 17:43:33.175168 5532 tablet_service.cc:1467] Processing CreateTablet for tablet 390f2fcb624840a2b187ef80c1a59650 (DEFAULT_TABLE table=test-table [id=f6a086b2b9b94e838e03883ebf45605a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:33.175468 5532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390f2fcb624840a2b187ef80c1a59650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.175420 5399 tablet_service.cc:1467] Processing CreateTablet for tablet 390f2fcb624840a2b187ef80c1a59650 (DEFAULT_TABLE table=test-table [id=f6a086b2b9b94e838e03883ebf45605a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:33.175750 5399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390f2fcb624840a2b187ef80c1a59650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.177495 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.169538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.1.8.129:42173)
0924 17:43:33.169725 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.169731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.169782 (+ 51us) server_negotiation.cc:407] Connection header received
0924 17:43:33.169828 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.169830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.169846 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.169930 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:33.170248 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.171580 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.173590 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.174188 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.174276 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.174331 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.174503 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.177402 (+ 2899us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260924 17:43:33.178192 5615 tablet_bootstrap.cc:492] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Bootstrap starting.
I20260924 17:43:33.178148 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 390f2fcb624840a2b187ef80c1a59650 (DEFAULT_TABLE table=test-table [id=f6a086b2b9b94e838e03883ebf45605a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:33.178516 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390f2fcb624840a2b187ef80c1a59650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.178520 5616 tablet_bootstrap.cc:492] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Bootstrap starting.
I20260924 17:43:33.179013 5615 tablet_bootstrap.cc:654] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.179132 5616 tablet_bootstrap.cc:654] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.179565 5615 log.cc:824] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.179752 5616 log.cc:824] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.180469 5616 tablet_bootstrap.cc:492] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: No bootstrap required, opened a new log
I20260924 17:43:33.180490 5615 tablet_bootstrap.cc:492] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: No bootstrap required, opened a new log
I20260924 17:43:33.180567 5616 ts_tablet_manager.cc:1397] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:43:33.180601 5615 ts_tablet_manager.cc:1397] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:43:33.182149 5619 tablet_bootstrap.cc:492] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd: Bootstrap starting.
I20260924 17:43:33.182376 5616 raft_consensus.cc:348] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.182632 5616 raft_consensus.cc:374] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.182729 5616 raft_consensus.cc:729] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c513b942352c4cca9810c07a3877427d, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.182869 5616 consensus_queue.cc:260] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.182806 5615 raft_consensus.cc:348] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.182963 5615 raft_consensus.cc:374] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.183030 5615 raft_consensus.cc:729] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659e2af60e244076a0d7e8b325cdd150, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.183125 5619 tablet_bootstrap.cc:654] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.183184 5616 ts_tablet_manager.cc:1428] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:43:33.183148 5615 consensus_queue.cc:260] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.183337 5465 heartbeater.cc:503] Master 127.1.8.190:36119 was elected leader, sending a full tablet report...
I20260924 17:43:33.183423 5615 ts_tablet_manager.cc:1428] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:33.183622 5598 heartbeater.cc:503] Master 127.1.8.190:36119 was elected leader, sending a full tablet report...
I20260924 17:43:33.183923 5619 log.cc:824] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.184803 5619 tablet_bootstrap.cc:492] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd: No bootstrap required, opened a new log
I20260924 17:43:33.184875 5619 ts_tablet_manager.cc:1397] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:33.186641 5619 raft_consensus.cc:348] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.186776 5619 raft_consensus.cc:374] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.186797 5619 raft_consensus.cc:729] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670269c1d35c471294c06768ebc0d5bd, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.186918 5619 consensus_queue.cc:260] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.187224 5619 ts_tablet_manager.cc:1428] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:43:33.187256 5332 heartbeater.cc:503] Master 127.1.8.190:36119 was elected leader, sending a full tablet report...
I20260924 17:43:33.216250 5620 raft_consensus.cc:482] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:33.216414 5620 raft_consensus.cc:504] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.216869 5620 leader_election.cc:288] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 659e2af60e244076a0d7e8b325cdd150 (127.1.8.131:36277), 670269c1d35c471294c06768ebc0d5bd (127.1.8.129:42173)
I20260924 17:43:33.220052 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.217238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:43775 (local address 127.1.8.129:42173)
0924 17:43:33.217293 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.217296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.217425 (+ 129us) server_negotiation.cc:407] Connection header received
0924 17:43:33.217473 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.217474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.217488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.217548 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:33.217738 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.218215 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.219258 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.219678 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.219765 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.219775 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.219932 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.219970 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:43:33.220073 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.217039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:36277 (local address 127.1.8.130:47021)
0924 17:43:33.217094 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.217111 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.217157 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.217381 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.217382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.217383 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:33.217477 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.217479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.218205 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.218206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.219087 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.219090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.219755 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.219756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.219855 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.219858 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.219932 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.219978 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 17:43:33.220084 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.217162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:47021 (local address 127.1.8.131:36277)
0924 17:43:33.217224 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.217227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.217238 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:33.217269 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.217270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.217283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.217376 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:33.217569 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.218110 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.219186 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.219675 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.219754 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.219768 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.219950 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.219990 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:43:33.220079 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.217143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42173 (local address 127.1.8.130:43775)
0924 17:43:33.217339 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.217350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.217412 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.217568 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.217569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.217570 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:33.217661 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.217663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.218299 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.218300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.219119 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:33.219121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.219775 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.219776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:33.219850 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.219853 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.219897 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.219961 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":123,"threads_started":1}
I20260924 17:43:33.220510 5552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390f2fcb624840a2b187ef80c1a59650" candidate_uuid: "c513b942352c4cca9810c07a3877427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659e2af60e244076a0d7e8b325cdd150" is_pre_election: true
I20260924 17:43:33.220710 5552 raft_consensus.cc:2454] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c513b942352c4cca9810c07a3877427d in term 0.
I20260924 17:43:33.220896 5286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390f2fcb624840a2b187ef80c1a59650" candidate_uuid: "c513b942352c4cca9810c07a3877427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670269c1d35c471294c06768ebc0d5bd" is_pre_election: true
I20260924 17:43:33.220976 5353 leader_election.cc:302] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659e2af60e244076a0d7e8b325cdd150, c513b942352c4cca9810c07a3877427d; no voters:
I20260924 17:43:33.221091 5286 raft_consensus.cc:2454] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c513b942352c4cca9810c07a3877427d in term 0.
I20260924 17:43:33.221107 5620 raft_consensus.cc:2781] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:33.221158 5620 raft_consensus.cc:482] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:33.221189 5620 raft_consensus.cc:3037] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:33.221946 5620 raft_consensus.cc:504] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.222064 5620 leader_election.cc:288] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [CANDIDATE]: Term 1 election: Requested vote from peers 659e2af60e244076a0d7e8b325cdd150 (127.1.8.131:36277), 670269c1d35c471294c06768ebc0d5bd (127.1.8.129:42173)
I20260924 17:43:33.222319 5552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390f2fcb624840a2b187ef80c1a59650" candidate_uuid: "c513b942352c4cca9810c07a3877427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659e2af60e244076a0d7e8b325cdd150"
I20260924 17:43:33.222337 5286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390f2fcb624840a2b187ef80c1a59650" candidate_uuid: "c513b942352c4cca9810c07a3877427d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670269c1d35c471294c06768ebc0d5bd"
I20260924 17:43:33.222433 5552 raft_consensus.cc:3037] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:33.222464 5286 raft_consensus.cc:3037] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:33.223133 5552 raft_consensus.cc:2454] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c513b942352c4cca9810c07a3877427d in term 1.
I20260924 17:43:33.223206 5286 raft_consensus.cc:2454] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c513b942352c4cca9810c07a3877427d in term 1.
I20260924 17:43:33.223373 5353 leader_election.cc:302] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 670269c1d35c471294c06768ebc0d5bd, c513b942352c4cca9810c07a3877427d; no voters:
I20260924 17:43:33.223493 5620 raft_consensus.cc:2781] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:33.223668 5620 raft_consensus.cc:686] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 1 LEADER]: Becoming Leader. State: Replica: c513b942352c4cca9810c07a3877427d, State: Running, Role: LEADER
I20260924 17:43:33.223784 5620 consensus_queue.cc:237] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } }
I20260924 17:43:33.224684 5141 catalog_manager.cc:5495] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d reported cstate change: term changed from 0 to 1, leader changed from <none> to c513b942352c4cca9810c07a3877427d (127.1.8.130). New cstate: current_term: 1 leader_uuid: "c513b942352c4cca9810c07a3877427d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c513b942352c4cca9810c07a3877427d" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 } health_report { overall_health: UNKNOWN } } }
W20260924 17:43:33.225986 5466 tablet.cc:2369] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:43:33.232640 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.229934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.1.8.130:40171)
0924 17:43:33.229995 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.229997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.230007 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:33.230136 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.230137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.230165 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.230247 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:33.230427 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.230972 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.231782 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:33.231982 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:33.232063 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:33.232253 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:33.232363 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:33.232465 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.232484 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.232515 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:43:33.234865 5286 raft_consensus.cc:1261] T 390f2fcb624840a2b187ef80c1a59650 P 670269c1d35c471294c06768ebc0d5bd [term 1 FOLLOWER]: Refusing update from remote peer c513b942352c4cca9810c07a3877427d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:33.234875 5552 raft_consensus.cc:1261] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 1 FOLLOWER]: Refusing update from remote peer c513b942352c4cca9810c07a3877427d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:33.235318 5620 consensus_queue.cc:1035] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "670269c1d35c471294c06768ebc0d5bd" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:33.235528 5625 consensus_queue.cc:1035] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [LEADER]: Connected to new peer: Peer: permanent_uuid: "659e2af60e244076a0d7e8b325cdd150" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:33.238585 5629 mvcc.cc:205] Tried to move back new op lower bound from 7332953347006484480 to 7332953346965442560. Current Snapshot: MvccSnapshot[applied={T|T < 7332953347006484480}]
I20260924 17:43:33.239068 5628 mvcc.cc:205] Tried to move back new op lower bound from 7332953347006484480 to 7332953346965442560. Current Snapshot: MvccSnapshot[applied={T|T < 7332953347006484480 or (T in {7332953347006484480})}]
I20260924 17:43:33.240266 5142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:33.240371 5142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:33.241220 5142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6275462e23d4ed5959f054521d6fb8c: Sending DeleteTablet for 3 replicas of tablet 390f2fcb624840a2b187ef80c1a59650
I20260924 17:43:33.241575 5532 tablet_service.cc:1514] Processing DeleteTablet for tablet 390f2fcb624840a2b187ef80c1a59650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:33 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:45080
I20260924 17:43:33.241598 5266 tablet_service.cc:1514] Processing DeleteTablet for tablet 390f2fcb624840a2b187ef80c1a59650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:33 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:44534
I20260924 17:43:33.241784 5399 tablet_service.cc:1514] Processing DeleteTablet for tablet 390f2fcb624840a2b187ef80c1a59650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:33 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:43240
I20260924 17:43:33.241986 5641 tablet_replica.cc:330] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: stopping tablet replica
I20260924 17:43:33.242085 5641 raft_consensus.cc:2229] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:33.242216 5641 raft_consensus.cc:2258] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:33.242235 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5204
I20260924 17:43:33.244232 5639 tablet_replica.cc:330] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: stopping tablet replica
I20260924 17:43:33.244359 5639 raft_consensus.cc:2229] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:33.244480 5639 raft_consensus.cc:2258] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:33.244927 5639 ts_tablet_manager.cc:1905] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:33.245002 5641 ts_tablet_manager.cc:1905] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:33.246979 5641 ts_tablet_manager.cc:1918] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:33.246966 5639 ts_tablet_manager.cc:1918] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:33.247064 5641 log.cc:1196] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/390f2fcb624840a2b187ef80c1a59650
I20260924 17:43:33.247083 5639 log.cc:1196] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/390f2fcb624840a2b187ef80c1a59650
I20260924 17:43:33.247368 5639 ts_tablet_manager.cc:1939] T 390f2fcb624840a2b187ef80c1a59650 P 659e2af60e244076a0d7e8b325cdd150: Deleting consensus metadata
I20260924 17:43:33.247368 5641 ts_tablet_manager.cc:1939] T 390f2fcb624840a2b187ef80c1a59650 P c513b942352c4cca9810c07a3877427d: Deleting consensus metadata
I20260924 17:43:33.247915 5128 catalog_manager.cc:4841] TS c513b942352c4cca9810c07a3877427d (127.1.8.130:40171): tablet 390f2fcb624840a2b187ef80c1a59650 (table test-table [id=f6a086b2b9b94e838e03883ebf45605a]) successfully deleted
I20260924 17:43:33.247967 5127 catalog_manager.cc:4841] TS 659e2af60e244076a0d7e8b325cdd150 (127.1.8.131:36277): tablet 390f2fcb624840a2b187ef80c1a59650 (table test-table [id=f6a086b2b9b94e838e03883ebf45605a]) successfully deleted
W20260924 17:43:33.253535 5127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:42173 (error 108)
I20260924 17:43:33.253854 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5337
I20260924 17:43:33.255293 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.255101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42173 (local address 127.0.0.1:44538)
0924 17:43:33.255183 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.255242 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:42173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 17:43:33.255775 5127 catalog_manager.cc:4584] TS 670269c1d35c471294c06768ebc0d5bd (127.1.8.129:42173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 390f2fcb624840a2b187ef80c1a59650: Network error: Client connection negotiation failed: client connection to 127.1.8.129:42173: connect: Connection refused (error 111)
I20260924 17:43:33.259173 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5470
I20260924 17:43:33.263880 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5111
2026-09-24T17:43:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2013 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T17:43:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:33Z Disabled control of system clock
I20260924 17:43:33.292379 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:46257
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 17:43:33.379127 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:33.379362 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:33.379385 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:33.380573 5646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:33.380625 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:33.380638 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:33.380651 5646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:33.380664 5646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:33.382011 5646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:46257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:33.382226 5646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:33.382578 5646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:33.387709 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.387683 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.387683 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:33.387717 5646 server_base.cc:1034] running on GCE node
I20260924 17:43:33.388163 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:33.388464 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:33.389624 5646 hybrid_clock.cc:648] HybridClock initialized: now 1790271813389585 us; error 42 us; skew 500 ppm
I20260924 17:43:33.391052 5646 webserver.cc:460] Webserver started at http://127.1.8.190:46221/ using document root <none> and password file <none>
I20260924 17:43:33.391321 5646 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:33.391394 5646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:33.391548 5646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:33.392707 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "d84e330b48634ca2beffb676890d6d5c"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "4b2bdf73fb93222243c4784788b36361"
server_key_iv: "cfbb7d2a9f1733f5c18ace7a591d2af2"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.393181 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "d84e330b48634ca2beffb676890d6d5c"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "4b2bdf73fb93222243c4784788b36361"
server_key_iv: "cfbb7d2a9f1733f5c18ace7a591d2af2"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.394630 5646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:33.395560 5660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.395875 5646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:33.395994 5646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "d84e330b48634ca2beffb676890d6d5c"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "4b2bdf73fb93222243c4784788b36361"
server_key_iv: "cfbb7d2a9f1733f5c18ace7a591d2af2"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.396091 5646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:33.399845 5646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:33.400159 5646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:33.400310 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:33.404340 5646 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:46257
I20260924 17:43:33.404374 5712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:46257 every 8 connection(s)
I20260924 17:43:33.404744 5646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:33.405439 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.407361 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5646
I20260924 17:43:33.407483 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:33.407675 1058 external_mini_cluster.cc:1428] Setting key 6101f559d1b9080869ee526da299494b
I20260924 17:43:33.409024 5713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c: Bootstrap starting.
I20260924 17:43:33.409822 5713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.410172 5713 log.cc:824] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.410390 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.408276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.1.8.190:46257)
0924 17:43:33.408481 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.408489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.408514 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:43:33.408915 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.408923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.408966 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.409089 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.409795 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.409798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.409798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.409802 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.409843 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.409854 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.409969 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.410056 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.410143 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260924 17:43:33.411049 5713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c: No bootstrap required, opened a new log
I20260924 17:43:33.412449 5713 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84e330b48634ca2beffb676890d6d5c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46257 } }
I20260924 17:43:33.412628 5713 raft_consensus.cc:374] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.412676 5713 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84e330b48634ca2beffb676890d6d5c, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.412810 5713 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84e330b48634ca2beffb676890d6d5c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46257 } }
I20260924 17:43:33.412906 5713 raft_consensus.cc:388] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:33.412949 5713 raft_consensus.cc:482] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:33.412999 5713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:33.413851 5713 raft_consensus.cc:504] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84e330b48634ca2beffb676890d6d5c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46257 } }
I20260924 17:43:33.413980 5713 leader_election.cc:302] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d84e330b48634ca2beffb676890d6d5c; no voters:
I20260924 17:43:33.414153 5713 leader_election.cc:288] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:33.414269 5718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:33.414479 5713 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:33.414479 5718 raft_consensus.cc:686] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [term 1 LEADER]: Becoming Leader. State: Replica: d84e330b48634ca2beffb676890d6d5c, State: Running, Role: LEADER
I20260924 17:43:33.414671 5718 consensus_queue.cc:237] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84e330b48634ca2beffb676890d6d5c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46257 } }
I20260924 17:43:33.415094 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d84e330b48634ca2beffb676890d6d5c. Latest consensus state: current_term: 1 leader_uuid: "d84e330b48634ca2beffb676890d6d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84e330b48634ca2beffb676890d6d5c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46257 } } }
I20260924 17:43:33.415211 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:33.415079 5719 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d84e330b48634ca2beffb676890d6d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d84e330b48634ca2beffb676890d6d5c" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46257 } } }
I20260924 17:43:33.415524 5719 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:33.415663 5727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:33.416189 5727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:33.417651 5727 catalog_manager.cc:1347] Generated new cluster ID: 6e309693218c42b0848961b326f29492
I20260924 17:43:33.417706 5727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:33.424336 5727 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:33.424562 5727 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:33.430356 5727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c: Generated new TSK 0
I20260924 17:43:33.430634 5727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:33.441764 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46257
--builtin_ntp_servers=127.1.8.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 17:43:33.526612 5737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:33.526849 5737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:33.526875 5737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:33.528187 5737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:33.528235 5737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:33.528264 5737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:33.529727 5737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:33.530046 5737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:33.530382 5737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:33.536909 5745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.536911 5743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.536911 5742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:33.537462 5737 server_base.cc:1034] running on GCE node
I20260924 17:43:33.537674 5737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:33.537933 5737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:33.539095 5737 hybrid_clock.cc:648] HybridClock initialized: now 1790271813539075 us; error 44 us; skew 500 ppm
I20260924 17:43:33.540448 5737 webserver.cc:460] Webserver started at http://127.1.8.129:41127/ using document root <none> and password file <none>
I20260924 17:43:33.540719 5737 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:33.540793 5737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:33.540946 5737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:33.542109 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "07e6909362dc4e038c075be566cf0f48"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "f41b684c063c3834c10cf2da992c807a"
server_key_iv: "937fd577363c401d1d6c94f1579f5207"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.542554 5737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "07e6909362dc4e038c075be566cf0f48"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "f41b684c063c3834c10cf2da992c807a"
server_key_iv: "937fd577363c401d1d6c94f1579f5207"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.543999 5737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:33.544858 5751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.545009 5737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:33.545115 5737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "07e6909362dc4e038c075be566cf0f48"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "f41b684c063c3834c10cf2da992c807a"
server_key_iv: "937fd577363c401d1d6c94f1579f5207"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.545212 5737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:33.548822 5737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:33.549124 5737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:33.549268 5737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:33.549475 5737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:33.549796 5737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:33.549829 5737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.549881 5737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:33.549911 5737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.555814 5737 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:33965
I20260924 17:43:33.555889 5864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:33965 every 8 connection(s)
I20260924 17:43:33.556249 5737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:43:33.556958 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5737
I20260924 17:43:33.557061 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:33.557279 1058 external_mini_cluster.cc:1428] Setting key de3142662c16121eeb26d8f0b306aa50
I20260924 17:43:33.558373 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46257
--builtin_ntp_servers=127.1.8.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 17:43:33.559876 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.557128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:36261 (local address 127.1.8.190:46257)
0924 17:43:33.557179 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.557182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.557554 (+ 372us) server_negotiation.cc:407] Connection header received
0924 17:43:33.558203 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.558204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.558212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.558282 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.559614 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.559624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.559625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.559630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.559651 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.559656 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.559716 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.559811 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.559823 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:43:33.560139 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.557137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:46257 (local address 127.1.8.129:36261)
0924 17:43:33.557367 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.557414 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.558073 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.558343 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.558347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.558356 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.559404 (+ 1048us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:33.559404 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.559436 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:33.559499 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.559728 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.559734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.559737 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.559823 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.559897 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260924 17:43:33.560729 5865 heartbeater.cc:347] Connected to a master server at 127.1.8.190:46257
I20260924 17:43:33.560837 5865 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:33.560961 5865 heartbeater.cc:511] Master 127.1.8.190:46257 requested a full tablet report, sending...
I20260924 17:43:33.561465 5677 ts_manager.cc:194] Registered new tserver with Master: 07e6909362dc4e038c075be566cf0f48 (127.1.8.129:33965)
W20260924 17:43:33.643056 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:33.643298 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:33.643321 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:33.644627 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:33.644672 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:33.644701 5868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:33.646116 5868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:33.646420 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:33.646785 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:33.652320 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.652398 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.652320 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:33.652487 5868 server_base.cc:1034] running on GCE node
I20260924 17:43:33.652809 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:33.653069 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:33.654247 5868 hybrid_clock.cc:648] HybridClock initialized: now 1790271813654230 us; error 27 us; skew 500 ppm
I20260924 17:43:33.655706 5868 webserver.cc:460] Webserver started at http://127.1.8.130:35285/ using document root <none> and password file <none>
I20260924 17:43:33.655982 5868 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:33.656054 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:33.656229 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:33.657367 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "fc790c37648a461e872b7b6116464f34"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "e38ea63e9a588bb67d90dc025bb297bd"
server_key_iv: "c38cbe12fe13a15ae8c701c7acb7269e"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.657737 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "fc790c37648a461e872b7b6116464f34"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "e38ea63e9a588bb67d90dc025bb297bd"
server_key_iv: "c38cbe12fe13a15ae8c701c7acb7269e"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.659237 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:33.660105 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.660362 5868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:33.660460 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "fc790c37648a461e872b7b6116464f34"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "e38ea63e9a588bb67d90dc025bb297bd"
server_key_iv: "c38cbe12fe13a15ae8c701c7acb7269e"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.660543 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:33.664182 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:33.664475 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:33.664624 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:33.664847 5868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:33.665170 5868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:33.665223 5868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.665263 5868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:33.665289 5868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.671077 5868 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:33895
I20260924 17:43:33.671162 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:33895 every 8 connection(s)
I20260924 17:43:33.671506 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:33.674175 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5868
I20260924 17:43:33.674292 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:33.674553 1058 external_mini_cluster.cc:1428] Setting key c9a48c14b072a19c57baf6287198bd97
I20260924 17:43:33.674739 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.672312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:35187 (local address 127.1.8.190:46257)
0924 17:43:33.672396 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.672399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.672622 (+ 223us) server_negotiation.cc:407] Connection header received
0924 17:43:33.673083 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.673084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.673089 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.673153 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.674328 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.674330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.674330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.674336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.674352 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.674357 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.674469 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.674647 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.674665 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 17:43:33.675009 5998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.672272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:46257 (local address 127.1.8.130:35187)
0924 17:43:33.672476 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.672520 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.672994 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.673207 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.673210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.673219 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.674138 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:33.674138 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.674168 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:33.674199 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.674483 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.674489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.674558 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.674658 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.674726 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260924 17:43:33.675459 5996 heartbeater.cc:347] Connected to a master server at 127.1.8.190:46257
I20260924 17:43:33.675560 5996 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:33.675586 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46257
--builtin_ntp_servers=127.1.8.148:45437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 17:43:33.675791 5996 heartbeater.cc:511] Master 127.1.8.190:46257 requested a full tablet report, sending...
I20260924 17:43:33.676203 5677 ts_manager.cc:194] Registered new tserver with Master: fc790c37648a461e872b7b6116464f34 (127.1.8.130:33895)
W20260924 17:43:33.758275 5999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:33.758528 5999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:33.758553 5999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:33.759801 5999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:33.759850 5999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:33.759877 5999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:33.761305 5999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:45437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:33.761582 5999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:33.761922 5999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:33.767143 6004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.767174 6005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:33.767130 6007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:33.767251 5999 server_base.cc:1034] running on GCE node
I20260924 17:43:33.767606 5999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:33.767863 5999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:33.769030 5999 hybrid_clock.cc:648] HybridClock initialized: now 1790271813769023 us; error 35 us; skew 500 ppm
I20260924 17:43:33.770442 5999 webserver.cc:460] Webserver started at http://127.1.8.131:46749/ using document root <none> and password file <none>
I20260924 17:43:33.770804 5999 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:33.770867 5999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:33.770973 5999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:33.772122 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "0c4157a52d63404b99fbc69a7c468c13"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "3777cea2c818e74f0759f7eec4db8af9"
server_key_iv: "67f488d3a06650ac3dccf580b9ba0168"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.772426 5999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "0c4157a52d63404b99fbc69a7c468c13"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "3777cea2c818e74f0759f7eec4db8af9"
server_key_iv: "67f488d3a06650ac3dccf580b9ba0168"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.773684 5999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:33.774385 6013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.774602 5999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:33.774677 5999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "0c4157a52d63404b99fbc69a7c468c13"
format_stamp: "Formatted at 2026-09-24 17:43:33 on dist-test-slave-r7hf"
server_key: "3777cea2c818e74f0759f7eec4db8af9"
server_key_iv: "67f488d3a06650ac3dccf580b9ba0168"
server_key_version: "encryptionkey@0"
I20260924 17:43:33.774729 5999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:33.778244 5999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:33.778548 5999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:33.778653 5999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:33.778817 5999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:33.779065 5999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:33.779094 5999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.779115 5999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:33.779129 5999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:33.785151 5999 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:34711
I20260924 17:43:33.785190 6126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:34711 every 8 connection(s)
I20260924 17:43:33.785571 5999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:43:33.788978 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.786568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:45471 (local address 127.1.8.190:46257)
0924 17:43:33.786634 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.786636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.786880 (+ 244us) server_negotiation.cc:407] Connection header received
0924 17:43:33.787419 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.787420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.787427 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.787480 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.788703 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.788705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.788706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.788711 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.788727 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.788732 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.788785 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.788876 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.788891 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:43:33.789189 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.786552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:46257 (local address 127.1.8.131:45471)
0924 17:43:33.786756 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.786791 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.787296 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.787651 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.787654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.787663 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.788545 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:33.788546 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.788575 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:33.788604 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.788782 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.788787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.788791 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.788888 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.788959 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260924 17:43:33.789719 6127 heartbeater.cc:347] Connected to a master server at 127.1.8.190:46257
I20260924 17:43:33.789799 6127 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:33.789920 6127 heartbeater.cc:511] Master 127.1.8.190:46257 requested a full tablet report, sending...
I20260924 17:43:33.790241 5677 ts_manager.cc:194] Registered new tserver with Master: 0c4157a52d63404b99fbc69a7c468c13 (127.1.8.131:34711)
I20260924 17:43:33.790597 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 5999
I20260924 17:43:33.790711 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:33.790897 1058 external_mini_cluster.cc:1428] Setting key 1d5de488e232cd652d73ddc4eef1a0d3
I20260924 17:43:33.791949 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:43:33.794401 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.793281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58952 (local address 127.1.8.190:46257)
0924 17:43:33.793342 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.793344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.793497 (+ 153us) server_negotiation.cc:407] Connection header received
0924 17:43:33.793545 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.793550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.793556 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.793597 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.794197 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.794198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.794198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.794201 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.794211 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.794214 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.794255 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.794286 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.794296 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:43:33.796180 5677 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:33.797767 5677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:33.802505 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.800841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43582 (local address 127.1.8.131:34711)
0924 17:43:33.801052 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.801057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.801097 (+ 40us) server_negotiation.cc:407] Connection header received
0924 17:43:33.801153 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.801156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.801166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.801257 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.802024 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.802033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.802034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.802039 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.802085 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.802112 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.802197 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.802337 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.802363 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260924 17:43:33.802536 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.800383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.1.8.130:33895)
0924 17:43:33.800550 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.800555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.800850 (+ 295us) server_negotiation.cc:407] Connection header received
0924 17:43:33.800965 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.800969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.800981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.801037 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.802171 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.802179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.802179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.802184 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.802219 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.802244 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.802280 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.802435 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.802451 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260924 17:43:33.802477 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.800569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34711 (local address 127.0.0.1:43582)
0924 17:43:33.801039 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.801048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.801094 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.801269 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.801274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.801281 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.801863 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:33.801863 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.801878 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:33.801919 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.802199 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.802207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.802210 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.802326 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.802351 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":115,"threads_started":1}
I20260924 17:43:33.802775 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.800195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:33895 (local address 127.0.0.1:33140)
0924 17:43:33.800786 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.800795 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.800871 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.801396 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.801396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.801399 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.802051 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:33.802051 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.802059 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:33.802070 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.802329 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.802329 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.802330 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.802403 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.802446 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260924 17:43:33.803155 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.800195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33965 (local address 127.0.0.1:36450)
0924 17:43:33.800549 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:33.800593 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:33.800688 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:33.801280 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:33.801281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:33.801285 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:33.802721 (+ 1436us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:33.802721 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:33.802729 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:33.802745 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:33.803027 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:33.803028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:33.803029 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:43:33.803084 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:43:33.803098 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":104,"threads_started":1}
I20260924 17:43:33.803241 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:33.800290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.1.8.129:33965)
0924 17:43:33.800515 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:33.800522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:33.800696 (+ 174us) server_negotiation.cc:407] Connection header received
0924 17:43:33.800770 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:33.800774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:33.800786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:33.800876 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:33.802861 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:33.802873 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:33.802874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:33.802879 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:33.802930 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:33.802965 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:33.803037 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:33.803095 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 17:43:33.803114 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260924 17:43:33.803556 5930 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6916371d5f43d5b67d4cfbb5db187f (DEFAULT_TABLE table=test-table [id=b2d6370a3c1444bfa44c436d417a09a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:33.804001 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6916371d5f43d5b67d4cfbb5db187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.803923 5799 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6916371d5f43d5b67d4cfbb5db187f (DEFAULT_TABLE table=test-table [id=b2d6370a3c1444bfa44c436d417a09a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:33.804260 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6916371d5f43d5b67d4cfbb5db187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.804239 6061 tablet_service.cc:1467] Processing CreateTablet for tablet 1d6916371d5f43d5b67d4cfbb5db187f (DEFAULT_TABLE table=test-table [id=b2d6370a3c1444bfa44c436d417a09a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:33.804657 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d6916371d5f43d5b67d4cfbb5db187f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:33.807623 6141 tablet_bootstrap.cc:492] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Bootstrap starting.
I20260924 17:43:33.807880 6142 tablet_bootstrap.cc:492] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: Bootstrap starting.
I20260924 17:43:33.807978 6143 tablet_bootstrap.cc:492] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Bootstrap starting.
I20260924 17:43:33.808763 6143 tablet_bootstrap.cc:654] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.808799 6142 tablet_bootstrap.cc:654] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.809158 6142 log.cc:824] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.809158 6143 log.cc:824] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.810015 6142 tablet_bootstrap.cc:492] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: No bootstrap required, opened a new log
I20260924 17:43:33.810102 6142 ts_tablet_manager.cc:1397] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:43:33.810200 6141 tablet_bootstrap.cc:654] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:33.810305 6143 tablet_bootstrap.cc:492] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: No bootstrap required, opened a new log
I20260924 17:43:33.810436 6143 ts_tablet_manager.cc:1397] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:33.810621 6141 log.cc:824] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:33.811532 6141 tablet_bootstrap.cc:492] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: No bootstrap required, opened a new log
I20260924 17:43:33.811618 6141 ts_tablet_manager.cc:1397] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 17:43:33.812458 6142 raft_consensus.cc:348] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:33.812633 6142 raft_consensus.cc:374] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.812677 6142 raft_consensus.cc:729] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e6909362dc4e038c075be566cf0f48, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.812816 6142 consensus_queue.cc:260] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:33.813097 6142 ts_tablet_manager.cc:1428] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:43:33.813108 5865 heartbeater.cc:503] Master 127.1.8.190:46257 was elected leader, sending a full tablet report...
I20260924 17:43:33.814049 6143 raft_consensus.cc:348] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:33.814284 6143 raft_consensus.cc:374] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.814339 6143 raft_consensus.cc:729] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4157a52d63404b99fbc69a7c468c13, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.814584 6143 consensus_queue.cc:260] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:33.814896 6143 ts_tablet_manager.cc:1428] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260924 17:43:33.814960 6127 heartbeater.cc:503] Master 127.1.8.190:46257 was elected leader, sending a full tablet report...
I20260924 17:43:33.815049 6141 raft_consensus.cc:348] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:33.815237 6141 raft_consensus.cc:374] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:33.815284 6141 raft_consensus.cc:729] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc790c37648a461e872b7b6116464f34, State: Initialized, Role: FOLLOWER
I20260924 17:43:33.815420 6141 consensus_queue.cc:260] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:33.815753 6141 ts_tablet_manager.cc:1428] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260924 17:43:33.815776 5996 heartbeater.cc:503] Master 127.1.8.190:46257 was elected leader, sending a full tablet report...
W20260924 17:43:33.922103 5997 tablet.cc:2369] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:43:34.036410 6128 tablet.cc:2369] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:43:34.056951 5866 tablet.cc:2369] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:43:34.202965 6147 raft_consensus.cc:482] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:34.203094 6147 raft_consensus.cc:504] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:34.203423 6147 leader_election.cc:288] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c4157a52d63404b99fbc69a7c468c13 (127.1.8.131:34711), fc790c37648a461e872b7b6116464f34 (127.1.8.130:33895)
I20260924 17:43:34.204988 6140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.203787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:45463 (local address 127.1.8.131:34711)
0924 17:43:34.203842 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.203844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.203886 (+ 42us) server_negotiation.cc:407] Connection header received
0924 17:43:34.203943 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.203944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.203952 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.204011 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:34.204752 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:34.204753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:34.204754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:34.204758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:34.204775 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:34.204779 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:34.204841 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:34.204907 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 17:43:34.204920 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:43:34.204988 6138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.203861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:50965 (local address 127.1.8.130:33895)
0924 17:43:34.203905 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.203907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.203947 (+ 40us) server_negotiation.cc:407] Connection header received
0924 17:43:34.203970 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.203971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.203975 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.204028 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:34.204752 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:34.204753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:34.204754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:34.204758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:34.204774 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:34.204779 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:34.204841 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:34.204896 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:43:34.204912 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:34.205044 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.203691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34711 (local address 127.1.8.129:45463)
0924 17:43:34.203823 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:34.203832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:34.203883 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:34.204052 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:34.204053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:34.204056 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:34.204648 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:34.204649 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:34.204653 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:34.204662 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:34.204844 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:34.204845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:34.204846 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:43:34.204895 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:43:34.204927 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":18,"thread_start_us":58,"threads_started":1}
I20260924 17:43:34.205044 6150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.203690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:33895 (local address 127.1.8.129:50965)
0924 17:43:34.203816 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:34.203828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:34.203882 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:34.204052 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:34.204053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:34.204071 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:34.204592 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:43:34.204592 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:34.204616 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:43:34.204634 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:34.204844 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:34.204845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:34.204846 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:43:34.204897 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:43:34.204921 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 17:43:34.205404 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6916371d5f43d5b67d4cfbb5db187f" candidate_uuid: "07e6909362dc4e038c075be566cf0f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4157a52d63404b99fbc69a7c468c13" is_pre_election: true
I20260924 17:43:34.205566 6081 raft_consensus.cc:2454] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e6909362dc4e038c075be566cf0f48 in term 0.
I20260924 17:43:34.205536 5950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6916371d5f43d5b67d4cfbb5db187f" candidate_uuid: "07e6909362dc4e038c075be566cf0f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc790c37648a461e872b7b6116464f34" is_pre_election: true
I20260924 17:43:34.205679 5950 raft_consensus.cc:2454] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07e6909362dc4e038c075be566cf0f48 in term 0.
I20260924 17:43:34.205794 5755 leader_election.cc:302] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e6909362dc4e038c075be566cf0f48, 0c4157a52d63404b99fbc69a7c468c13; no voters:
I20260924 17:43:34.205976 6147 raft_consensus.cc:2781] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:34.206017 6147 raft_consensus.cc:482] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:34.206033 6147 raft_consensus.cc:3037] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:34.206795 6147 raft_consensus.cc:504] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:34.206959 6147 leader_election.cc:288] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [CANDIDATE]: Term 1 election: Requested vote from peers 0c4157a52d63404b99fbc69a7c468c13 (127.1.8.131:34711), fc790c37648a461e872b7b6116464f34 (127.1.8.130:33895)
I20260924 17:43:34.207108 5950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6916371d5f43d5b67d4cfbb5db187f" candidate_uuid: "07e6909362dc4e038c075be566cf0f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc790c37648a461e872b7b6116464f34"
I20260924 17:43:34.207108 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d6916371d5f43d5b67d4cfbb5db187f" candidate_uuid: "07e6909362dc4e038c075be566cf0f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4157a52d63404b99fbc69a7c468c13"
I20260924 17:43:34.207185 5950 raft_consensus.cc:3037] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:34.207185 6081 raft_consensus.cc:3037] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:34.208070 6081 raft_consensus.cc:2454] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e6909362dc4e038c075be566cf0f48 in term 1.
I20260924 17:43:34.208071 5950 raft_consensus.cc:2454] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07e6909362dc4e038c075be566cf0f48 in term 1.
I20260924 17:43:34.208258 5755 leader_election.cc:302] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07e6909362dc4e038c075be566cf0f48, 0c4157a52d63404b99fbc69a7c468c13; no voters:
I20260924 17:43:34.208371 6147 raft_consensus.cc:2781] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:34.208515 6147 raft_consensus.cc:686] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 1 LEADER]: Becoming Leader. State: Replica: 07e6909362dc4e038c075be566cf0f48, State: Running, Role: LEADER
I20260924 17:43:34.208630 6147 consensus_queue.cc:237] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } }
I20260924 17:43:34.209434 5677 catalog_manager.cc:5495] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07e6909362dc4e038c075be566cf0f48 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "07e6909362dc4e038c075be566cf0f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07e6909362dc4e038c075be566cf0f48" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 } health_report { overall_health: UNKNOWN } } }
I20260924 17:43:34.247112 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.246013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.1.8.129:33965)
0924 17:43:34.246083 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.246085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.246093 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:34.246146 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.246147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.246161 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.246204 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:34.246904 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:34.246905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:34.246905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:34.246910 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:34.246926 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:34.246930 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:34.246993 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:34.247041 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 17:43:34.247053 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:43:34.249354 5950 raft_consensus.cc:1261] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 1 FOLLOWER]: Refusing update from remote peer 07e6909362dc4e038c075be566cf0f48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:34.249377 6081 raft_consensus.cc:1261] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 1 FOLLOWER]: Refusing update from remote peer 07e6909362dc4e038c075be566cf0f48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:34.249773 6147 consensus_queue.cc:1035] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4157a52d63404b99fbc69a7c468c13" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:34.249938 6152 consensus_queue.cc:1035] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc790c37648a461e872b7b6116464f34" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:34.252512 6155 mvcc.cc:205] Tried to move back new op lower bound from 7332953351162290176 to 7332953350999097344. Current Snapshot: MvccSnapshot[applied={T|T < 7332953351162290176}]
I20260924 17:43:34.255657 5676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:34.255787 5676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:34.256601 5676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d84e330b48634ca2beffb676890d6d5c: Sending DeleteTablet for 3 replicas of tablet 1d6916371d5f43d5b67d4cfbb5db187f
I20260924 17:43:34.256950 5930 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6916371d5f43d5b67d4cfbb5db187f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:34 UTC) from {username='slave'} at 127.0.0.1:33140
I20260924 17:43:34.256997 6061 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6916371d5f43d5b67d4cfbb5db187f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:34 UTC) from {username='slave'} at 127.0.0.1:43582
I20260924 17:43:34.257189 5799 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d6916371d5f43d5b67d4cfbb5db187f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:34 UTC) from {username='slave'} at 127.0.0.1:36450
I20260924 17:43:34.257256 6165 tablet_replica.cc:330] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: stopping tablet replica
I20260924 17:43:34.257390 6165 raft_consensus.cc:2229] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:34.257418 6166 tablet_replica.cc:330] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48: stopping tablet replica
I20260924 17:43:34.257509 6166 raft_consensus.cc:2229] T 1d6916371d5f43d5b67d4cfbb5db187f P 07e6909362dc4e038c075be566cf0f48 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:34.257622 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5737
I20260924 17:43:34.258600 6165 raft_consensus.cc:2258] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:34.258839 6164 tablet_replica.cc:330] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: stopping tablet replica
I20260924 17:43:34.258958 6164 raft_consensus.cc:2229] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:34.259066 6164 raft_consensus.cc:2258] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:34.259523 6164 ts_tablet_manager.cc:1905] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:34.260125 6165 ts_tablet_manager.cc:1905] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:34.261242 6164 ts_tablet_manager.cc:1918] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:34.261330 6164 log.cc:1196] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/1d6916371d5f43d5b67d4cfbb5db187f
I20260924 17:43:34.261600 6164 ts_tablet_manager.cc:1939] T 1d6916371d5f43d5b67d4cfbb5db187f P 0c4157a52d63404b99fbc69a7c468c13: Deleting consensus metadata
I20260924 17:43:34.262033 6165 ts_tablet_manager.cc:1918] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:34.262145 6165 log.cc:1196] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/1d6916371d5f43d5b67d4cfbb5db187f
I20260924 17:43:34.262295 5664 catalog_manager.cc:4841] TS 0c4157a52d63404b99fbc69a7c468c13 (127.1.8.131:34711): tablet 1d6916371d5f43d5b67d4cfbb5db187f (table test-table [id=b2d6370a3c1444bfa44c436d417a09a5]) successfully deleted
I20260924 17:43:34.262459 6165 ts_tablet_manager.cc:1939] T 1d6916371d5f43d5b67d4cfbb5db187f P fc790c37648a461e872b7b6116464f34: Deleting consensus metadata
I20260924 17:43:34.262923 5663 catalog_manager.cc:4841] TS fc790c37648a461e872b7b6116464f34 (127.1.8.130:33895): tablet 1d6916371d5f43d5b67d4cfbb5db187f (table test-table [id=b2d6370a3c1444bfa44c436d417a09a5]) successfully deleted
W20260924 17:43:34.263593 5664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:33965 (error 108)
I20260924 17:43:34.264189 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.263974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33965 (local address 127.0.0.1:45930)
0924 17:43:34.264061 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:34.264134 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 17:43:34.264209 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5868
W20260924 17:43:34.264353 5664 catalog_manager.cc:4584] TS 07e6909362dc4e038c075be566cf0f48 (127.1.8.129:33965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d6916371d5f43d5b67d4cfbb5db187f: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33965: connect: Connection refused (error 111)
I20260924 17:43:34.268973 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5999
I20260924 17:43:34.273447 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 5646
2026-09-24T17:43:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1008 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 17:43:34.293356 1058 mini_oidc.cc:152] Starting JWKS server
I20260924 17:43:34.294514 1058 webserver.cc:460] Webserver started at https://127.0.0.1:39367/ using document root <none> and password file <none>
I20260924 17:43:34.294638 1058 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 17:43:34.294848 1058 webserver.cc:460] Webserver started at http://127.0.0.1:35751/ using document root <none> and password file <none>
2026-09-24T17:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:34Z Disabled control of system clock
I20260924 17:43:34.302860 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:42577
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:42577
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39367/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/testchainca.cert with env {}
W20260924 17:43:34.389546 6173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:34.389773 6173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:34.389796 6173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:34.391153 6173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:34.391209 6173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:34.391222 6173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:34.391235 6173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:34.391288 6173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:43:34.391307 6173 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:43:34.391330 6173 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39367/jwks.json
I20260924 17:43:34.393069 6173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:42577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:42577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39367/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:34.393376 6173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:34.393647 6173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:34.398772 6179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:34.398772 6178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:34.398808 6181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:34.399353 6173 server_base.cc:1034] running on GCE node
I20260924 17:43:34.399626 6173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:34.399945 6173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:34.401125 6173 hybrid_clock.cc:648] HybridClock initialized: now 1790271814401101 us; error 35 us; skew 500 ppm
I20260924 17:43:34.402640 6173 webserver.cc:460] Webserver started at http://127.1.8.190:42729/ using document root <none> and password file <none>
I20260924 17:43:34.402917 6173 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:34.402999 6173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:34.403149 6173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:34.404292 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "2db99d1894284b9f88d4c170682264f9"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "38de6c1eb8fb456077f52a9dcb7ae058"
server_key_iv: "3aff6c460912b2944177fe729b31788f"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.404685 6173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "2db99d1894284b9f88d4c170682264f9"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "38de6c1eb8fb456077f52a9dcb7ae058"
server_key_iv: "3aff6c460912b2944177fe729b31788f"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.406028 6173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:34.406878 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:34.407097 6173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:43:34.407223 6173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "2db99d1894284b9f88d4c170682264f9"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "38de6c1eb8fb456077f52a9dcb7ae058"
server_key_iv: "3aff6c460912b2944177fe729b31788f"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.407303 6173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:34.415109 6173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:34.415449 6173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:34.415606 6173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:34.419734 6173 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:42577
I20260924 17:43:34.419768 6239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:42577 every 8 connection(s)
I20260924 17:43:34.420137 6173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:34.421121 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:34.423601 6240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9: Bootstrap starting.
I20260924 17:43:34.424247 6240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:34.424553 6240 log.cc:824] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:34.425315 6240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9: No bootstrap required, opened a new log
I20260924 17:43:34.427013 6240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db99d1894284b9f88d4c170682264f9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 42577 } }
I20260924 17:43:34.427166 6240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:34.427191 6240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db99d1894284b9f88d4c170682264f9, State: Initialized, Role: FOLLOWER
I20260924 17:43:34.427363 6240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db99d1894284b9f88d4c170682264f9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 42577 } }
I20260924 17:43:34.427444 6240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:34.427479 6240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:34.427526 6240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:34.427642 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6173
I20260924 17:43:34.427733 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:34.427899 1058 external_mini_cluster.cc:1428] Setting key 12f4463492d16f4a5ddf00b7e150ca72
I20260924 17:43:34.428261 6240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db99d1894284b9f88d4c170682264f9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 42577 } }
I20260924 17:43:34.428378 6240 leader_election.cc:302] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2db99d1894284b9f88d4c170682264f9; no voters:
I20260924 17:43:34.428597 6240 leader_election.cc:288] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:34.428668 6245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:34.428831 6245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2db99d1894284b9f88d4c170682264f9, State: Running, Role: LEADER
I20260924 17:43:34.428946 6240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:34.428963 6245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db99d1894284b9f88d4c170682264f9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 42577 } }
I20260924 17:43:34.429569 6246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2db99d1894284b9f88d4c170682264f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db99d1894284b9f88d4c170682264f9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 42577 } } }
I20260924 17:43:34.429665 6246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:34.429891 6247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2db99d1894284b9f88d4c170682264f9. Latest consensus state: current_term: 1 leader_uuid: "2db99d1894284b9f88d4c170682264f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2db99d1894284b9f88d4c170682264f9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 42577 } } }
I20260924 17:43:34.429994 6247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9 [sys.catalog]: This master's current role is: LEADER
W20260924 17:43:34.430909 6260 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:34.430997 6260 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:34.431087 6254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:34.431241 6254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:34.432973 6254 catalog_manager.cc:1347] Generated new cluster ID: 7f90d8d54f4546df992ccc4115490b14
I20260924 17:43:34.433024 6254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:34.434581 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.428423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.1.8.190:42577)
0924 17:43:34.428698 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.428709 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.428744 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:43:34.428879 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.428887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.428962 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.429424 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:34.429535 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.431089 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.432244 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.432901 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.433073 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.433750 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:34.433753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:34.433754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:34.433952 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:34.433995 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:34.434017 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:34.434078 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:34.434180 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 17:43:34.434342 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":141,"threads_started":1}
I20260924 17:43:34.435510 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:34.461412 6254 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:34.461979 6254 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:34.468627 6254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2db99d1894284b9f88d4c170682264f9: Generated new TSK 0
I20260924 17:43:34.468853 6254 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 17:43:34.501402 6270 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 17:43:34.501529 6270 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 17:43:34.501912 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.485633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.1.8.190:42577)
0924 17:43:34.485701 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.485703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.485826 (+ 123us) server_negotiation.cc:407] Connection header received
0924 17:43:34.485908 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.485909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.485922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.486007 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:43:34.486154 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.486901 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.488041 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.488371 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.488414 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.488674 (+ 260us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:43:34.501673 (+ 12999us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 17:43:34.501841 (+ 168us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 17:43:34.501974 6244 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 17:43:34.502264 6269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:34.485525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:42577 (local address 127.0.0.1:50660)
0924 17:43:34.485756 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:34.485766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:34.485828 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:34.486018 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:34.486018 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:34.486020 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:43:34.486086 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:34.486088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.486984 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.486985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:34.487932 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:34.487935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.488470 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.488474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:34.488523 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.488542 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:43:34.501825 (+ 13283us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 17:43:34.501921 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.8.190:42577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":114,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.8.190:42577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39367/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 17:43:34.502884 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6173
2026-09-24T17:43:34Z chronyd exiting
I20260924 17:43:34.539398 1058 test_util.cc:175] -----------------------------------------------
I20260924 17:43:34.539482 1058 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790271791394214-1058-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 17:43:34.540207 1058 mini_oidc.cc:152] Starting JWKS server
I20260924 17:43:34.540987 1058 webserver.cc:460] Webserver started at https://127.0.0.1:33253/ using document root <none> and password file <none>
I20260924 17:43:34.541033 1058 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 17:43:34.541180 1058 webserver.cc:460] Webserver started at http://127.0.0.1:35261/ using document root <none> and password file <none>
2026-09-24T17:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:34Z Disabled control of system clock
I20260924 17:43:34.549150 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:34645
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:37053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:34645
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/testchainca.cert with env {}
W20260924 17:43:34.632319 6276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:34.632612 6276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:34.632653 6276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:34.633929 6276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:34.634009 6276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:34.634034 6276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:34.634069 6276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:34.634099 6276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:43:34.634123 6276 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:43:34.634147 6276 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33253/jwks.json
I20260924 17:43:34.635787 6276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:34645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:34.636093 6276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:34.636333 6276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:34.641552 6284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:34.641552 6281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:34.641582 6282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:34.641848 6276 server_base.cc:1034] running on GCE node
I20260924 17:43:34.642127 6276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:34.642472 6276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:34.643664 6276 hybrid_clock.cc:648] HybridClock initialized: now 1790271814643665 us; error 47 us; skew 500 ppm
I20260924 17:43:34.645112 6276 webserver.cc:460] Webserver started at http://127.1.8.190:34599/ using document root <none> and password file <none>
I20260924 17:43:34.645390 6276 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:34.645558 6276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:34.645725 6276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:34.646934 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "ec6ff1642f0441e98b294069e9eedeb5"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "92f9cb87cca3126b90b1da6817fc286a"
server_key_iv: "dfa1b9a2a2799d8139111b7902781bd9"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.647303 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "ec6ff1642f0441e98b294069e9eedeb5"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "92f9cb87cca3126b90b1da6817fc286a"
server_key_iv: "dfa1b9a2a2799d8139111b7902781bd9"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.648716 6276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:34.649560 6290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:34.649771 6276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:34.649887 6276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "ec6ff1642f0441e98b294069e9eedeb5"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "92f9cb87cca3126b90b1da6817fc286a"
server_key_iv: "dfa1b9a2a2799d8139111b7902781bd9"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.649971 6276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:34.670585 6276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:34.670956 6276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:34.671120 6276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:34.675150 6276 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:34645
I20260924 17:43:34.675194 6342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:34645 every 8 connection(s)
I20260924 17:43:34.675586 6276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:34.676720 6343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:34.679385 6343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5: Bootstrap starting.
I20260924 17:43:34.679961 6343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:34.680207 6343 log.cc:824] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:34.680917 6343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5: No bootstrap required, opened a new log
I20260924 17:43:34.682753 6343 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6ff1642f0441e98b294069e9eedeb5" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34645 } }
I20260924 17:43:34.682915 6343 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:34.682941 6343 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec6ff1642f0441e98b294069e9eedeb5, State: Initialized, Role: FOLLOWER
I20260924 17:43:34.683060 6343 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6ff1642f0441e98b294069e9eedeb5" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34645 } }
I20260924 17:43:34.683127 6343 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:34.683157 6343 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:34.683187 6343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:34.683727 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6276
I20260924 17:43:34.683840 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:34.683797 6343 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6ff1642f0441e98b294069e9eedeb5" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34645 } }
I20260924 17:43:34.683941 6343 leader_election.cc:302] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec6ff1642f0441e98b294069e9eedeb5; no voters:
I20260924 17:43:34.684073 1058 external_mini_cluster.cc:1428] Setting key b8d3e1ade6893841ba9bf0423dd60240
I20260924 17:43:34.684209 6343 leader_election.cc:288] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:34.684249 6346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:34.684510 6346 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [term 1 LEADER]: Becoming Leader. State: Replica: ec6ff1642f0441e98b294069e9eedeb5, State: Running, Role: LEADER
I20260924 17:43:34.684659 6343 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:34.684661 6346 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6ff1642f0441e98b294069e9eedeb5" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34645 } }
I20260924 17:43:34.685063 6347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec6ff1642f0441e98b294069e9eedeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6ff1642f0441e98b294069e9eedeb5" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34645 } } }
I20260924 17:43:34.685042 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec6ff1642f0441e98b294069e9eedeb5. Latest consensus state: current_term: 1 leader_uuid: "ec6ff1642f0441e98b294069e9eedeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec6ff1642f0441e98b294069e9eedeb5" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 34645 } } }
I20260924 17:43:34.685159 6347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:34.685164 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:34.685487 6354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:34.685983 6354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:34.687712 6354 catalog_manager.cc:1347] Generated new cluster ID: 39d7168e9f974df5831eadef04ab06e8
I20260924 17:43:34.687764 6354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:34.690495 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.684808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.1.8.190:34645)
0924 17:43:34.685064 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.685074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.685109 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:43:34.685252 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.685261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.685355 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.685538 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:34.685774 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.687049 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.688193 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.688784 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.688912 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.689735 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:34.689739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:34.689740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:34.689948 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:34.689988 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:34.690002 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:34.690064 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:34.690188 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 17:43:34.690266 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260924 17:43:34.691476 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:34.702355 6354 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:34.703048 6354 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:34.711549 6354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec6ff1642f0441e98b294069e9eedeb5: Generated new TSK 0
I20260924 17:43:34.711833 6354 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 17:43:34.744342 6373 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 17:43:34.744450 6373 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 17:43:34.744771 6350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.723381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.1.8.190:34645)
0924 17:43:34.723436 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.723438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.723543 (+ 105us) server_negotiation.cc:407] Connection header received
0924 17:43:34.723634 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.723634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.723645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.723710 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:43:34.723886 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.724404 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.725619 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.725827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.725887 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.726110 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:43:34.744487 (+ 18377us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 17:43:34.744713 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 17:43:34.744853 6350 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 17:43:34.744876 6372 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:34.723268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:34645 (local address 127.0.0.1:54646)
0924 17:43:34.723456 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:34.723470 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:34.723550 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:34.723738 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:34.723740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:34.723742 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:43:34.723814 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:34.723817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.724558 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.724559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:34.725507 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:34.725511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.725925 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.725929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:34.725994 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.726001 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:43:34.744644 (+ 18643us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 17:43:34.744767 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.8.190:34645: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33253/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260924 17:43:34.745223 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6276
2026-09-24T17:43:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (242 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 17:43:34.783344 1058 mini_oidc.cc:152] Starting JWKS server
I20260924 17:43:34.784070 1058 webserver.cc:460] Webserver started at https://127.0.0.1:33785/ using document root <none> and password file <none>
I20260924 17:43:34.784178 1058 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 17:43:34.784364 1058 webserver.cc:460] Webserver started at http://127.0.0.1:38421/ using document root <none> and password file <none>
2026-09-24T17:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:34Z Disabled control of system clock
I20260924 17:43:34.792400 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:33203
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:35689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:33203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33785/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 17:43:34.875898 6379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:34.876156 6379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:34.876181 6379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:34.877558 6379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:34.877619 6379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:34.877637 6379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:34.877660 6379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:34.877671 6379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:43:34.877684 6379 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:43:34.877699 6379 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33785/jwks.json
I20260924 17:43:34.879415 6379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:35689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:33203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:33203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33785/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:34.879683 6379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:34.879971 6379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:34.885787 6384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:34.885844 6379 server_base.cc:1034] running on GCE node
W20260924 17:43:34.885854 6385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:34.885787 6387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:34.886277 6379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:34.886627 6379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:34.887805 6379 hybrid_clock.cc:648] HybridClock initialized: now 1790271814887793 us; error 46 us; skew 500 ppm
I20260924 17:43:34.889460 6379 webserver.cc:460] Webserver started at http://127.1.8.190:34691/ using document root <none> and password file <none>
I20260924 17:43:34.889747 6379 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:34.889798 6379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:34.889890 6379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:34.891098 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "f57c823328394d7586b5666e910ae5a9"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "acf1f49c8a32884b04c36e1990f85fb1"
server_key_iv: "c946b12dc45d28911a35f4f42a9f5fc2"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.891408 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "f57c823328394d7586b5666e910ae5a9"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "acf1f49c8a32884b04c36e1990f85fb1"
server_key_iv: "c946b12dc45d28911a35f4f42a9f5fc2"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.892711 6379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:34.893433 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:34.893643 6379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:34.893750 6379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "f57c823328394d7586b5666e910ae5a9"
format_stamp: "Formatted at 2026-09-24 17:43:34 on dist-test-slave-r7hf"
server_key: "acf1f49c8a32884b04c36e1990f85fb1"
server_key_iv: "c946b12dc45d28911a35f4f42a9f5fc2"
server_key_version: "encryptionkey@0"
I20260924 17:43:34.893829 6379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:34.904740 6379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:34.905104 6379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:34.905251 6379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:34.909406 6379 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:33203
I20260924 17:43:34.909571 6445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:33203 every 8 connection(s)
I20260924 17:43:34.909838 6379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:34.910642 6446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:34.913865 6446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9: Bootstrap starting.
I20260924 17:43:34.914505 6446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:34.914764 6446 log.cc:824] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:34.915467 6446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9: No bootstrap required, opened a new log
I20260924 17:43:34.917047 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6379
I20260924 17:43:34.917135 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:34.917078 6446 raft_consensus.cc:348] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57c823328394d7586b5666e910ae5a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33203 } }
I20260924 17:43:34.917299 6446 raft_consensus.cc:374] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:34.917351 1058 external_mini_cluster.cc:1428] Setting key 86dbdeb6a018a2612ee94433bad2759b
I20260924 17:43:34.917348 6446 raft_consensus.cc:729] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57c823328394d7586b5666e910ae5a9, State: Initialized, Role: FOLLOWER
I20260924 17:43:34.917517 6446 consensus_queue.cc:260] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57c823328394d7586b5666e910ae5a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33203 } }
I20260924 17:43:34.917600 6446 raft_consensus.cc:388] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:34.917644 6446 raft_consensus.cc:482] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:34.917691 6446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:34.918478 6446 raft_consensus.cc:504] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57c823328394d7586b5666e910ae5a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33203 } }
I20260924 17:43:34.918592 6446 leader_election.cc:302] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f57c823328394d7586b5666e910ae5a9; no voters:
I20260924 17:43:34.918757 6446 leader_election.cc:288] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:34.918941 6452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:34.919054 6452 raft_consensus.cc:686] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [term 1 LEADER]: Becoming Leader. State: Replica: f57c823328394d7586b5666e910ae5a9, State: Running, Role: LEADER
I20260924 17:43:34.919068 6446 sys_catalog.cc:564] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:34.919147 6452 consensus_queue.cc:237] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57c823328394d7586b5666e910ae5a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33203 } }
I20260924 17:43:34.919468 6452 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f57c823328394d7586b5666e910ae5a9. Latest consensus state: current_term: 1 leader_uuid: "f57c823328394d7586b5666e910ae5a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57c823328394d7586b5666e910ae5a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33203 } } }
I20260924 17:43:34.919579 6452 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:34.919480 6453 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f57c823328394d7586b5666e910ae5a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f57c823328394d7586b5666e910ae5a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33203 } } }
I20260924 17:43:34.919718 6453 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:34.919783 6458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:34.920327 6458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:34.922091 6458 catalog_manager.cc:1347] Generated new cluster ID: d8ba945355ce4825817a6c0b8b5bffbf
I20260924 17:43:34.922143 6458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:34.924144 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.917858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46252 (local address 127.1.8.190:33203)
0924 17:43:34.918164 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.918174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.918209 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:43:34.918335 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.918343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.918475 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.918658 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:34.918977 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.920824 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.921910 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.922530 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.922651 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.923409 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:34.923413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:34.923414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:34.923631 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:34.923675 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:34.923691 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:34.923762 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:34.923832 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 17:43:34.923914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260924 17:43:34.925112 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:34.958689 6458 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:34.959281 6458 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:34.965131 6458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f57c823328394d7586b5666e910ae5a9: Generated new TSK 0
I20260924 17:43:34.965382 6458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:34.975963 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:34.972174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.1.8.190:33203)
0924 17:43:34.972235 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:34.972237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:34.972338 (+ 101us) server_negotiation.cc:407] Connection header received
0924 17:43:34.972395 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:34.972396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:34.972409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:34.972496 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:43:34.972631 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.973395 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.974172 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.974508 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.974581 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.974760 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:43:34.975748 (+ 988us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 17:43:34.975888 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 17:43:34.976029 6450 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46268: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 17:43:34.977947 6475 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:34.972071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:33203 (local address 127.0.0.1:46268)
0924 17:43:34.972268 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:34.972278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:34.972336 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:34.972506 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:34.972507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:34.972509 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:43:34.972562 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:34.972564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.973481 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.973482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:34.974083 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:34.974084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:34.974596 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:34.974604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:34.974649 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:34.974655 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:43:34.975857 (+ 1202us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 17:43:34.977844 (+ 1987us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.8.190:33203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33785/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1766,"thread_start_us":117,"threads_started":1}
I20260924 17:43:34.978317 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6379
E20260924 17:43:34.987565 6476 webserver.cc:533] Webserver: sslize failed:
E20260924 17:43:34.987671 6476 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T17:43:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T17:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:35Z Disabled control of system clock
I20260924 17:43:35.025451 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45957
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45957
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/testchainca.cert with env {}
W20260924 17:43:35.116921 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:35.117179 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:35.117205 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:35.118527 6480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:35.118583 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:35.118599 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:35.118649 6480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:35.118681 6480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 17:43:35.118706 6480 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 17:43:35.118739 6480 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 17:43:35.120469 6480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:35.120749 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:35.120991 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:35.126191 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:35.126230 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:35.126191 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:35.126490 6480 server_base.cc:1034] running on GCE node
I20260924 17:43:35.126765 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:35.127090 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:35.128280 6480 hybrid_clock.cc:648] HybridClock initialized: now 1790271815128264 us; error 26 us; skew 500 ppm
I20260924 17:43:35.129612 6480 webserver.cc:460] Webserver started at http://127.1.8.190:45293/ using document root <none> and password file <none>
I20260924 17:43:35.129863 6480 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:35.129935 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:35.130106 6480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:35.131292 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "a2847f44397c4d92a57c753c4b1bdc09"
format_stamp: "Formatted at 2026-09-24 17:43:35 on dist-test-slave-r7hf"
server_key: "75c283db176b1a5dc7758b8306792867"
server_key_iv: "e0aa2fc95d49142a5c0d83749186627d"
server_key_version: "encryptionkey@0"
I20260924 17:43:35.131606 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "a2847f44397c4d92a57c753c4b1bdc09"
format_stamp: "Formatted at 2026-09-24 17:43:35 on dist-test-slave-r7hf"
server_key: "75c283db176b1a5dc7758b8306792867"
server_key_iv: "e0aa2fc95d49142a5c0d83749186627d"
server_key_version: "encryptionkey@0"
I20260924 17:43:35.132900 6480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:43:35.133623 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:35.133781 6480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:43:35.133853 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "a2847f44397c4d92a57c753c4b1bdc09"
format_stamp: "Formatted at 2026-09-24 17:43:35 on dist-test-slave-r7hf"
server_key: "75c283db176b1a5dc7758b8306792867"
server_key_iv: "e0aa2fc95d49142a5c0d83749186627d"
server_key_version: "encryptionkey@0"
I20260924 17:43:35.133908 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:35.171094 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:35.171406 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:35.171514 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:35.175577 6480 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:45957
I20260924 17:43:35.175660 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:45957 every 8 connection(s)
I20260924 17:43:35.175988 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:35.176975 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:35.179497 6547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09: Bootstrap starting.
I20260924 17:43:35.180059 6547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:35.180303 6547 log.cc:824] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:35.180989 6547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09: No bootstrap required, opened a new log
I20260924 17:43:35.181022 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6480
I20260924 17:43:35.181126 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:35.181322 1058 external_mini_cluster.cc:1428] Setting key 5fe8a9f13d413077ed5fa1a92c53024d
I20260924 17:43:35.183449 6547 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2847f44397c4d92a57c753c4b1bdc09" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45957 } }
I20260924 17:43:35.183665 6547 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:35.183709 6547 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2847f44397c4d92a57c753c4b1bdc09, State: Initialized, Role: FOLLOWER
I20260924 17:43:35.183851 6547 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2847f44397c4d92a57c753c4b1bdc09" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45957 } }
I20260924 17:43:35.183940 6547 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:35.183976 6547 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:35.184020 6547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:35.184764 6547 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2847f44397c4d92a57c753c4b1bdc09" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45957 } }
I20260924 17:43:35.184870 6547 leader_election.cc:302] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2847f44397c4d92a57c753c4b1bdc09; no voters:
I20260924 17:43:35.185011 6547 leader_election.cc:288] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:35.185101 6552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:35.185256 6547 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:35.185266 6552 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [term 1 LEADER]: Becoming Leader. State: Replica: a2847f44397c4d92a57c753c4b1bdc09, State: Running, Role: LEADER
I20260924 17:43:35.186326 6552 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2847f44397c4d92a57c753c4b1bdc09" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45957 } }
I20260924 17:43:35.187075 6554 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2847f44397c4d92a57c753c4b1bdc09. Latest consensus state: current_term: 1 leader_uuid: "a2847f44397c4d92a57c753c4b1bdc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2847f44397c4d92a57c753c4b1bdc09" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45957 } } }
I20260924 17:43:35.187132 6554 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:35.186946 6553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2847f44397c4d92a57c753c4b1bdc09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2847f44397c4d92a57c753c4b1bdc09" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45957 } } }
I20260924 17:43:35.187284 6553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:35.187501 6567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:35.187915 6567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:35.188875 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:35.181869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.1.8.190:45957)
0924 17:43:35.182121 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:35.182131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:35.182166 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:43:35.183002 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:35.183012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:35.183106 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:35.183276 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:35.183525 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:35.185237 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:35.186202 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:35.186935 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:35.187066 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:35.188113 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:35.188117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:35.188118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:43:35.188329 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:35.188379 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:43:35.188395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:35.188472 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:35.188571 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 17:43:35.188659 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260924 17:43:35.189925 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:43:35.190047 6567 catalog_manager.cc:1347] Generated new cluster ID: 790c74047f5d4cf699e1b99f60a1c48d
I20260924 17:43:35.190083 6567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:35.198256 6567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:35.198906 6567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:35.206848 6567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2847f44397c4d92a57c753c4b1bdc09: Generated new TSK 0
I20260924 17:43:35.207127 6567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:43:35.340415 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:35.213083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.1.8.190:45957)
0924 17:43:35.213123 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:35.213126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:35.213253 (+ 127us) server_negotiation.cc:407] Connection header received
0924 17:43:35.213356 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:35.213357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:35.213369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:35.213468 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0924 17:43:35.213599 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:35.214132 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:35.215022 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:35.215267 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:35.215357 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:35.215620 (+ 263us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 17:43:35.340104 (+124484us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 17:43:35.340319 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:43:35.340510 6551 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52560: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 17:43:35.340512 6576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:35.213016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:45957 (local address 127.0.0.1:52560)
0924 17:43:35.213172 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:35.213184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:35.213273 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:35.213467 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:35.213468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:35.213470 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 17:43:35.213532 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:35.213534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:35.214230 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:35.214232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:35.214920 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:35.214922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:35.215361 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:35.215370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:35.215433 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:35.215439 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 17:43:35.340280 (+124841us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 17:43:35.340399 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.8.190:45957: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260924 17:43:35.340867 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6480
2026-09-24T17:43:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (342 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:35 dist-test-slave-r7hf krb5kdc[6588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:35 dist-test-slave-r7hf krb5kdc[6588](info): set up 2 sockets
Sep 24 17:43:35 dist-test-slave-r7hf krb5kdc[6588](info): commencing operation
krb5kdc: starting...
W20260924 17:43:36.558650 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:36 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:43:36.683646 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:43551
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:43551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:36.766417 6601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:36.766683 6601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:36.766721 6601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:36.767948 6601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:36.768030 6601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:36.768054 6601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:36.768074 6601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:36.768103 6601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:36.769574 6601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:43551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:43551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:36.769853 6601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:36.770093 6601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:36.775036 6609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:36.775048 6606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:36.775152 6607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:36.775578 6601 server_base.cc:1034] running on GCE node
I20260924 17:43:36.775839 6601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:36.776140 6601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:36.777297 6601 hybrid_clock.cc:648] HybridClock initialized: now 1790271816777294 us; error 39 us; skew 500 ppm
Sep 24 17:43:36 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271816, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:36.779827 6601 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:43:36.780280 6601 webserver.cc:460] Webserver started at http://127.1.8.190:33755/ using document root <none> and password file <none>
I20260924 17:43:36.780530 6601 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:36.780597 6601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:36.780738 6601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:36.781787 6601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "cac76b8799fc4c16b337adee50552cea"
format_stamp: "Formatted at 2026-09-24 17:43:36 on dist-test-slave-r7hf"
server_key: "76835afbadfea7d90d56d313aa20f8f9"
server_key_iv: "06e09622ce26c435ef48f4825b6dc518"
server_key_version: "encryptionkey@0"
I20260924 17:43:36.782158 6601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "cac76b8799fc4c16b337adee50552cea"
format_stamp: "Formatted at 2026-09-24 17:43:36 on dist-test-slave-r7hf"
server_key: "76835afbadfea7d90d56d313aa20f8f9"
server_key_iv: "06e09622ce26c435ef48f4825b6dc518"
server_key_version: "encryptionkey@0"
I20260924 17:43:36.783687 6601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:36.784472 6616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:36.784678 6601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:36.784791 6601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "cac76b8799fc4c16b337adee50552cea"
format_stamp: "Formatted at 2026-09-24 17:43:36 on dist-test-slave-r7hf"
server_key: "76835afbadfea7d90d56d313aa20f8f9"
server_key_iv: "06e09622ce26c435ef48f4825b6dc518"
server_key_version: "encryptionkey@0"
I20260924 17:43:36.784870 6601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:36.798166 6601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:36.800649 6601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:36.800822 6601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:36.804880 6601 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:43551
I20260924 17:43:36.804935 6668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:43551 every 8 connection(s)
I20260924 17:43:36.805320 6601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:36.806222 6669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:36.808562 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea: Bootstrap starting.
I20260924 17:43:36.808701 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6601
I20260924 17:43:36.808802 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:36.809026 1058 external_mini_cluster.cc:1428] Setting key 5ca970d187d48df3277cf939800ad2d3
I20260924 17:43:36.809298 6669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:36.809621 6669 log.cc:824] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:36.810365 6669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea: No bootstrap required, opened a new log
I20260924 17:43:36.812685 6669 raft_consensus.cc:348] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac76b8799fc4c16b337adee50552cea" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43551 } }
I20260924 17:43:36.812870 6669 raft_consensus.cc:374] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:36.812911 6669 raft_consensus.cc:729] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac76b8799fc4c16b337adee50552cea, State: Initialized, Role: FOLLOWER
I20260924 17:43:36.813072 6669 consensus_queue.cc:260] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac76b8799fc4c16b337adee50552cea" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43551 } }
I20260924 17:43:36.813153 6669 raft_consensus.cc:388] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:36.813192 6669 raft_consensus.cc:482] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:36.813241 6669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:36.813850 6669 raft_consensus.cc:504] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac76b8799fc4c16b337adee50552cea" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43551 } }
I20260924 17:43:36.813958 6669 leader_election.cc:302] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cac76b8799fc4c16b337adee50552cea; no voters:
I20260924 17:43:36.814119 6669 leader_election.cc:288] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:36.814181 6674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:36.814368 6674 raft_consensus.cc:686] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [term 1 LEADER]: Becoming Leader. State: Replica: cac76b8799fc4c16b337adee50552cea, State: Running, Role: LEADER
I20260924 17:43:36.814513 6674 consensus_queue.cc:237] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac76b8799fc4c16b337adee50552cea" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43551 } }
I20260924 17:43:36.814373 6669 sys_catalog.cc:564] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:36.814893 6676 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac76b8799fc4c16b337adee50552cea. Latest consensus state: current_term: 1 leader_uuid: "cac76b8799fc4c16b337adee50552cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac76b8799fc4c16b337adee50552cea" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43551 } } }
I20260924 17:43:36.814953 6675 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cac76b8799fc4c16b337adee50552cea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac76b8799fc4c16b337adee50552cea" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 43551 } } }
I20260924 17:43:36.815140 6675 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:36.815948 6678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:36.815026 6676 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:36.816447 6678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:43:36 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:36.818606 6678 catalog_manager.cc:1347] Generated new cluster ID: 9c89c7c8600a441793fcafe537e2eb85
I20260924 17:43:36.818665 6678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:36.820822 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:36.809609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.1.8.190:43551)
0924 17:43:36.809801 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:36.809811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:36.809848 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:43:36.810398 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:36.810433 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:36.810502 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:36.810666 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:36.811272 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:36.812748 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:36.813762 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:36.814475 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:36.814587 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:36.817383 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:36.817387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:36.817387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:36.817401 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:36.819318 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:36.819688 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:36.819691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:36.819692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:36.819775 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:36.819977 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:36.819978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:36.819978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:36.820218 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:36.820387 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:36.820478 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 17:43:36.820617 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260924 17:43:36.829893 6678 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:36.830678 6678 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:36.834769 6678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cac76b8799fc4c16b337adee50552cea: Generated new TSK 0
I20260924 17:43:36.834975 6678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:43:36.869642 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43551
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:36.989437 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:36.989764 6695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:36.989804 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:36.991855 6695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:36.991930 6695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:36.991976 6695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:36.994314 6695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:36.994626 6695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:36.994887 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:37.000425 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:37.000414 6700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:37.000414 6701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:37.000985 6695 server_base.cc:1034] running on GCE node
I20260924 17:43:37.001246 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:37.001507 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:37.002676 6695 hybrid_clock.cc:648] HybridClock initialized: now 1790271817002649 us; error 40 us; skew 500 ppm
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.005507 6695 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:43:37.006014 6695 webserver.cc:460] Webserver started at http://127.1.8.129:41411/ using document root <none> and password file <none>
I20260924 17:43:37.006227 6695 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:37.006269 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:37.006362 6695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:37.007532 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "2e92bfe47b5d43d5a4f2f1eefe21c24b"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "90cdf8900d9924ddf6331f71f1ca9807"
server_key_iv: "ee33a9388a3f44d6c8d80a7efceed112"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.007841 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "2e92bfe47b5d43d5a4f2f1eefe21c24b"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "90cdf8900d9924ddf6331f71f1ca9807"
server_key_iv: "ee33a9388a3f44d6c8d80a7efceed112"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.009146 6695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:37.009855 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.010025 6695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:37.010129 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "2e92bfe47b5d43d5a4f2f1eefe21c24b"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "90cdf8900d9924ddf6331f71f1ca9807"
server_key_iv: "ee33a9388a3f44d6c8d80a7efceed112"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.010205 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:37.047174 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:37.049341 6695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:37.049513 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:37.049765 6695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:37.050104 6695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:37.050168 6695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.050211 6695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:37.050235 6695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.056409 6695 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:35091
I20260924 17:43:37.056453 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:35091 every 8 connection(s)
I20260924 17:43:37.056792 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:37.066176 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.057468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:43551 (local address 127.1.8.129:34223)
0924 17:43:37.057638 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:37.057682 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:37.058686 (+ 1004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:37.058925 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:37.058929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:37.059203 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:37.059401 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:37.059405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.060080 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.060081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:37.060917 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:37.060919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.061282 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.061283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:37.061383 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.061877 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:37.061886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:37.063409 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:37.064940 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:37.064946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:37.064948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:37.065168 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:37.065347 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:37.065348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:37.065348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:37.065445 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:37.065671 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:37.065679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:37.065797 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 17:43:37.065881 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:43:37.066000 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260924 17:43:37.066227 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.057516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:34223 (local address 127.1.8.190:43551)
0924 17:43:37.057590 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.057592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.057773 (+ 181us) server_negotiation.cc:407] Connection header received
0924 17:43:37.058784 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.058785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.058795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.058861 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.059482 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.059987 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.061020 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.061210 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.061261 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.063518 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:37.063520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:37.063520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:37.063527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:37.064867 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.065235 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.065236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.065236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.065291 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.065509 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.065509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.065509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.065616 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:37.065667 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:37.065903 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 17:43:37.066109 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:43:37.066252 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6695
I20260924 17:43:37.066350 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:37.066588 1058 external_mini_cluster.cc:1428] Setting key bae7d2ba27b30ef7dc19355bdbe0b22d
I20260924 17:43:37.066974 6824 heartbeater.cc:347] Connected to a master server at 127.1.8.190:43551
I20260924 17:43:37.067101 6824 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:37.067284 6824 heartbeater.cc:511] Master 127.1.8.190:43551 requested a full tablet report, sending...
I20260924 17:43:37.067868 6633 ts_manager.cc:194] Registered new tserver with Master: 2e92bfe47b5d43d5a4f2f1eefe21c24b (127.1.8.129:35091)
I20260924 17:43:37.068776 6633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:34223
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:43:37.093601 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43551
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:37.176297 6829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:37.176573 6829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:37.176614 6829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:37.177837 6829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:37.177914 6829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:37.177963 6829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:37.179626 6829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:37.179926 6829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:37.180164 6829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:37.185307 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:37.185348 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:37.185319 6837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:37.185510 6829 server_base.cc:1034] running on GCE node
I20260924 17:43:37.185719 6829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:37.185966 6829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:37.187107 6829 hybrid_clock.cc:648] HybridClock initialized: now 1790271817187096 us; error 29 us; skew 500 ppm
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.189657 6829 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:43:37.190068 6829 webserver.cc:460] Webserver started at http://127.1.8.130:37403/ using document root <none> and password file <none>
I20260924 17:43:37.190280 6829 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:37.190321 6829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:37.190472 6829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:37.191530 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "4c954c64b1484f54b681b3432d09ae30"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "b30d5441d8c04cd01c8b8b41e0228eda"
server_key_iv: "474981f758e5ca04fb9a638767407e0b"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.191834 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "4c954c64b1484f54b681b3432d09ae30"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "b30d5441d8c04cd01c8b8b41e0228eda"
server_key_iv: "474981f758e5ca04fb9a638767407e0b"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.193133 6829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:37.193864 6844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.194043 6829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:37.194157 6829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "4c954c64b1484f54b681b3432d09ae30"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "b30d5441d8c04cd01c8b8b41e0228eda"
server_key_iv: "474981f758e5ca04fb9a638767407e0b"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.194242 6829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:37.222028 6829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:37.224298 6829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:37.224467 6829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:37.224720 6829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:37.225034 6829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:37.225095 6829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.225137 6829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:37.225162 6829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.231149 6829 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:41559
I20260924 17:43:37.231196 6957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:41559 every 8 connection(s)
I20260924 17:43:37.231560 6829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:37.239012 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6829
I20260924 17:43:37.239152 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:37.239352 1058 external_mini_cluster.cc:1428] Setting key 99277e6bf2ea66fa36a1a16bca08a4f0
I20260924 17:43:37.241379 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.232369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:34837 (local address 127.1.8.190:43551)
0924 17:43:37.232433 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.232434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.232676 (+ 242us) server_negotiation.cc:407] Connection header received
0924 17:43:37.233691 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.233692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.233704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.233763 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.234454 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.234976 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.236008 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.236219 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.236303 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.238440 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:37.238441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:37.238442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:37.238450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:37.239986 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.240419 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.240420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.240421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.240482 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.240731 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.240732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.240732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.240843 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:37.240911 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:37.241196 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 17:43:37.241282 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:43:37.241588 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.232350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:43551 (local address 127.1.8.130:34837)
0924 17:43:37.232548 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:37.232591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:37.233577 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:37.233841 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:37.233844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:37.234131 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:37.234330 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:37.234334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.235096 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.235098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:37.235905 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:37.235908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.236311 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.236312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:37.236433 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.236965 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:37.236973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:37.238318 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:37.240090 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:37.240098 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:37.240101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:37.240330 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:37.240548 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:37.240549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:37.240549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:37.240666 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:37.240915 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:37.240922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:37.241061 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 17:43:37.241176 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 17:43:37.241341 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
I20260924 17:43:37.242123 6958 heartbeater.cc:347] Connected to a master server at 127.1.8.190:43551
I20260924 17:43:37.242251 6958 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:37.242488 6958 heartbeater.cc:511] Master 127.1.8.190:43551 requested a full tablet report, sending...
I20260924 17:43:37.242908 6633 ts_manager.cc:194] Registered new tserver with Master: 4c954c64b1484f54b681b3432d09ae30 (127.1.8.130:41559)
I20260924 17:43:37.243288 6633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:34837
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:43:37.266350 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43551
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:37.352288 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:37.352527 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:37.352555 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:37.353843 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:37.353906 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:37.353936 6963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:37.355614 6963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:37.355894 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:37.356146 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:43:37.361486 6963 server_base.cc:1034] running on GCE node
W20260924 17:43:37.361402 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:37.361408 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:37.361411 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:37.361954 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:37.362221 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:37.363457 6963 hybrid_clock.cc:648] HybridClock initialized: now 1790271817363424 us; error 42 us; skew 500 ppm
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.366153 6963 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:43:37.366701 6963 webserver.cc:460] Webserver started at http://127.1.8.131:36507/ using document root <none> and password file <none>
I20260924 17:43:37.366967 6963 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:37.367035 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:37.367190 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:37.368336 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "679c8b0cc3014ccc85c861e257be5534"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "15dba88df87f7a1e91498c7143deef5f"
server_key_iv: "26c10cb1fa40646bb3038d034535c871"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.368711 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "679c8b0cc3014ccc85c861e257be5534"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "15dba88df87f7a1e91498c7143deef5f"
server_key_iv: "26c10cb1fa40646bb3038d034535c871"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.370095 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:43:37.370882 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.371033 6963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.371136 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "679c8b0cc3014ccc85c861e257be5534"
format_stamp: "Formatted at 2026-09-24 17:43:37 on dist-test-slave-r7hf"
server_key: "15dba88df87f7a1e91498c7143deef5f"
server_key_iv: "26c10cb1fa40646bb3038d034535c871"
server_key_version: "encryptionkey@0"
I20260924 17:43:37.371212 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:37.393354 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:37.395530 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:37.395709 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:37.395968 6963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:37.396288 6963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:37.396342 6963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.396386 6963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:37.396413 6963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:37.402424 6963 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:41873
I20260924 17:43:37.402482 7091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:41873 every 8 connection(s)
I20260924 17:43:37.402846 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:43:37.411480 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 6963
I20260924 17:43:37.411561 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:37.411787 1058 external_mini_cluster.cc:1428] Setting key 3ff182a7d2555034bb63a65b69f4c575
I20260924 17:43:37.413240 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.403673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:38935 (local address 127.1.8.190:43551)
0924 17:43:37.403738 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.403740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.403986 (+ 246us) server_negotiation.cc:407] Connection header received
0924 17:43:37.405003 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.405004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.405017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.405084 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.405717 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.406222 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.407409 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.407648 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.407710 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.410180 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:37.410182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:37.410182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:37.410191 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:37.411711 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.412230 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.412231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.412232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.412320 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.412562 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.412563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.412564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.412726 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:37.412798 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:37.413108 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 17:43:37.413176 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:37.413427 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.403640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:43551 (local address 127.1.8.131:38935)
0924 17:43:37.403809 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:37.403855 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:37.404875 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:37.405145 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:37.405148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:37.405421 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:37.405634 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:37.405638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.406315 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.406316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:37.407324 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:37.407327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.407762 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.407763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:37.407883 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.408567 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:37.408575 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:37.410051 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:37.411807 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:37.411815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:37.411818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:37.412152 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:37.412379 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:37.412380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:37.412380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:37.412516 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:37.412806 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:37.412812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:37.412967 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 17:43:37.413057 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 17:43:37.413185 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 17:43:37.414021 7092 heartbeater.cc:347] Connected to a master server at 127.1.8.190:43551
I20260924 17:43:37.414162 7092 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:37.414345 7092 heartbeater.cc:511] Master 127.1.8.190:43551 requested a full tablet report, sending...
I20260924 17:43:37.414757 6633 ts_manager.cc:194] Registered new tserver with Master: 679c8b0cc3014ccc85c861e257be5534 (127.1.8.131:41873)
I20260924 17:43:37.415211 6633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:38935
I20260924 17:43:37.423683 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/test-admin.keytab.
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.440033 1058 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 17:43:37.441759 7099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:37.440939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.1.8.129:35091)
0924 17:43:37.441109 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.441115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.441127 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:43:37.441162 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.441164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.441178 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.441253 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.441664 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57224: BlockingRecv error: recv got EOF from 127.0.0.1:57224 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
W20260924 17:43:37.441792 1058 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc)
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.442822 1058 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.443737 1058 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:37.452241 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.444348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.1.8.129:35091)
0924 17:43:37.444486 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.444488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.444497 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:37.444526 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.444526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.444539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.444608 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.445020 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.445772 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.446555 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.446903 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.446985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.449224 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:37.449249 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:37.449250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:37.449257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:37.450953 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.451354 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.451358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.451359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.451421 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.451646 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.451647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.451648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.451811 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:37.451990 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:37.452036 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 17:43:37.452119 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 17:43:37.454310 7099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:37.453634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.1.8.129:35091)
0924 17:43:37.453683 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.453685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.453758 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:43:37.453797 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.453798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.453807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.453852 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.454259 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57244: BlockingRecv error: recv got EOF from 127.0.0.1:57244 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 17:43:37.454360 1058 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc' of type FILE
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.455456 1058 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.456362 1058 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:37.463732 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.456938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.1.8.129:35091)
0924 17:43:37.457122 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.457124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.457135 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:37.457231 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.457232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.457248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.457301 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.457710 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.458194 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.458958 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.459164 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.459248 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.461327 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:37.461328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:37.461329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:37.461335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:37.462636 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.462956 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.462957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.462957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.463005 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.463165 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.463165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.463166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.463255 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:37.463326 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:37.463550 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 17:43:37.463619 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
W20260924 17:43:37.465252 7099 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:43:37.464777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.1.8.129:35091)
0924 17:43:37.464820 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.464822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.464850 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:43:37.464907 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.464908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.464914 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.464964 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.465198 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57252: BlockingRecv error: recv got EOF from 127.0.0.1:57252 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 17:43:37.465335 1058 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790271791394214-1058-0/krb5kdc/krb5cc)
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.466445 1058 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:37.467337 1058 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[6588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.129@KRBTEST.COM
I20260924 17:43:37.474299 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:37.467851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.1.8.129:35091)
0924 17:43:37.467909 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:37.467911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:37.467955 (+ 44us) server_negotiation.cc:407] Connection header received
0924 17:43:37.468037 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:37.468038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:37.468046 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:37.468122 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:37.468510 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.468975 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.469670 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:37.469851 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:37.469920 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:37.472108 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:37.472110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:37.472110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:37.472120 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:37.473294 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.473583 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.473584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.473585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.473630 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:37.473844 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:37.473844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:37.473845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:37.473940 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:37.474043 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:37.474124 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 17:43:37.474178 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:43:37.475127 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6695
I20260924 17:43:37.479698 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6829
I20260924 17:43:37.484202 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6963
I20260924 17:43:37.488780 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 6601
2026-09-24T17:43:37Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1970 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[7112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[7112](info): set up 2 sockets
Sep 24 17:43:37 dist-test-slave-r7hf krb5kdc[7112](info): commencing operation
krb5kdc: starting...
W20260924 17:43:38.590756 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:43:38 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:43:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab.
Entry for principal oryx/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab.
I20260924 17:43:38.717773 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39843
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab
--principal=oryx/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:38.800570 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:38.800849 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:38.800889 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:38.802208 7125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:38.802300 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:38.802323 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:38.802345 7125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:38.802378 7125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:38.804014 7125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab
--principal=oryx/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:38.804343 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:38.804589 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:38.809912 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:38.809912 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:38.810106 7125 server_base.cc:1034] running on GCE node
W20260924 17:43:38.809939 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:38.810487 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:38.810796 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:38.811941 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790271818811918 us; error 43 us; skew 500 ppm
Sep 24 17:43:38 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:38.819881 7125 init.cc:377] Logged in from keytab as oryx/127.1.8.190@KRBTEST.COM (short username oryx)
I20260924 17:43:38.820338 7125 webserver.cc:460] Webserver started at http://127.1.8.190:45089/ using document root <none> and password file <none>
I20260924 17:43:38.820550 7125 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:38.820596 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:38.820691 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:38.821766 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "d79e6ab0487943a99573b2a4e2c379d7"
format_stamp: "Formatted at 2026-09-24 17:43:38 on dist-test-slave-r7hf"
server_key: "dfc72403f09ed483b62b8cb9d754777f"
server_key_iv: "4c6b6e1a5be65e69f11441148be4f4aa"
server_key_version: "encryptionkey@0"
I20260924 17:43:38.822080 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "d79e6ab0487943a99573b2a4e2c379d7"
format_stamp: "Formatted at 2026-09-24 17:43:38 on dist-test-slave-r7hf"
server_key: "dfc72403f09ed483b62b8cb9d754777f"
server_key_iv: "4c6b6e1a5be65e69f11441148be4f4aa"
server_key_version: "encryptionkey@0"
I20260924 17:43:38.823714 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:43:38.824615 7140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:38.824810 7125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:38.824920 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "d79e6ab0487943a99573b2a4e2c379d7"
format_stamp: "Formatted at 2026-09-24 17:43:38 on dist-test-slave-r7hf"
server_key: "dfc72403f09ed483b62b8cb9d754777f"
server_key_iv: "4c6b6e1a5be65e69f11441148be4f4aa"
server_key_version: "encryptionkey@0"
I20260924 17:43:38.825001 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:38.832818 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:38.835016 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:38.835188 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:38.839289 7125 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39843
I20260924 17:43:38.839342 7192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39843 every 8 connection(s)
I20260924 17:43:38.839726 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:43:38.840781 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:38.842978 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7125
I20260924 17:43:38.843111 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:43:38.843331 1058 external_mini_cluster.cc:1428] Setting key f5ed0e29dab4fea99c01a693fd7e5d55
I20260924 17:43:38.843809 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Bootstrap starting.
I20260924 17:43:38.844533 7193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:38.844904 7193 log.cc:824] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:38.845759 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: No bootstrap required, opened a new log
I20260924 17:43:38.848255 7193 raft_consensus.cc:348] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } }
I20260924 17:43:38.848464 7193 raft_consensus.cc:374] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:38.848510 7193 raft_consensus.cc:729] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79e6ab0487943a99573b2a4e2c379d7, State: Initialized, Role: FOLLOWER
I20260924 17:43:38.848668 7193 consensus_queue.cc:260] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } }
I20260924 17:43:38.848781 7193 raft_consensus.cc:388] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:43:38.848816 7193 raft_consensus.cc:482] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:43:38.848874 7193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:38.849846 7193 raft_consensus.cc:504] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } }
I20260924 17:43:38.850000 7193 leader_election.cc:302] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d79e6ab0487943a99573b2a4e2c379d7; no voters:
I20260924 17:43:38.850209 7193 leader_election.cc:288] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:43:38.850267 7198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:38.850425 7198 raft_consensus.cc:686] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 1 LEADER]: Becoming Leader. State: Replica: d79e6ab0487943a99573b2a4e2c379d7, State: Running, Role: LEADER
I20260924 17:43:38.850525 7198 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } }
I20260924 17:43:38.850574 7193 sys_catalog.cc:564] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:38.850878 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } }
I20260924 17:43:38.850951 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: This master's current role is: LEADER
Sep 24 17:43:38 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:38.851085 7198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d79e6ab0487943a99573b2a4e2c379d7. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } }
I20260924 17:43:38.851145 7198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:38.852608 7202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:43:38.853420 7202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:43:38.855171 7202 catalog_manager.cc:1347] Generated new cluster ID: 11715045a2fc44c08f8c2b623651aa38
I20260924 17:43:38.855226 7202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:43:38.855726 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:38.843918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.1.8.190:39843)
0924 17:43:38.844100 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:38.844123 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:38.844150 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:43:38.844615 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:38.844624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:38.844688 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:38.844809 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:38.845675 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:38.846679 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:38.847876 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:38.848293 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:38.848382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:38.851703 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:38.851708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:38.851708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:38.851723 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:38.853715 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:38.854081 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:38.854085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:38.854085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:38.854178 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:38.854449 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:38.854450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:38.854450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:38.854765 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:38.854952 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:38.855473 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 17:43:38.855584 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260924 17:43:38.891716 7202 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:43:38.892500 7202 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:43:38.899081 7202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Generated new TSK 0
I20260924 17:43:38.899303 7202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab.
Entry for principal oryx/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab.
I20260924 17:43:38.934144 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39843
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:39.017724 7219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:39.017969 7219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:39.017999 7219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:39.019330 7219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:39.019381 7219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:39.019435 7219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:39.020977 7219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:39.021246 7219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:39.021500 7219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.026820 7224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.026916 7227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.026973 7219 server_base.cc:1034] running on GCE node
W20260924 17:43:39.026800 7225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.027287 7219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.027520 7219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.028712 7219 hybrid_clock.cc:648] HybridClock initialized: now 1790271819028690 us; error 37 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.031993 7219 init.cc:377] Logged in from keytab as oryx/127.1.8.129@KRBTEST.COM (short username oryx)
I20260924 17:43:39.032665 7219 webserver.cc:460] Webserver started at http://127.1.8.129:45941/ using document root <none> and password file <none>
I20260924 17:43:39.032954 7219 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.033016 7219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.033197 7219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:39.034353 7219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "ea480d5a2def493c81aac3c1599ef1d1"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "cb48ac273b5881ceb2321fead4486637"
server_key_iv: "feea395cf2e3dd7b0a4da2ca29c13b02"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.034817 7219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "ea480d5a2def493c81aac3c1599ef1d1"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "cb48ac273b5881ceb2321fead4486637"
server_key_iv: "feea395cf2e3dd7b0a4da2ca29c13b02"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.036517 7219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 17:43:39.037472 7234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.037660 7219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:39.037774 7219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "ea480d5a2def493c81aac3c1599ef1d1"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "cb48ac273b5881ceb2321fead4486637"
server_key_iv: "feea395cf2e3dd7b0a4da2ca29c13b02"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.037863 7219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.060652 7219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:39.063442 7219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:39.063584 7219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:39.063814 7219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:39.064181 7219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:39.064218 7219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.064246 7219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:39.064263 7219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.071508 7219 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:45267
I20260924 17:43:39.071558 7347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:45267 every 8 connection(s)
I20260924 17:43:39.071946 7219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:39.079591 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7219
I20260924 17:43:39.079698 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:43:39.079916 1058 external_mini_cluster.cc:1428] Setting key e162860d1172abe4981835c0fe624c1d
I20260924 17:43:39.082899 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.072793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:37039 (local address 127.1.8.190:39843)
0924 17:43:39.072862 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.072865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.073082 (+ 217us) server_negotiation.cc:407] Connection header received
0924 17:43:39.073637 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.073638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.073650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.073708 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.074795 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.075674 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.076699 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.077067 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.077138 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.079378 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.079379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.079380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.079388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.081315 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.081862 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.081864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.081864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.081933 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.082187 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.082188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.082189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.082306 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.082385 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.082711 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.082800 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:43:39.083137 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.072776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:37039)
0924 17:43:39.072953 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.072992 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.073518 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.073798 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.073801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.074463 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.074693 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.074698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.075782 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.075782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.076610 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.076612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.077159 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.077160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.077263 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.077835 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.077844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.079286 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.081404 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.081412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.081415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.081753 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.082021 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.082021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.082022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.082121 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.082375 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.082382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.082577 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.082691 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.082862 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260924 17:43:39.083793 7348 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39843
I20260924 17:43:39.083911 7348 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:39.084113 7348 heartbeater.cc:511] Master 127.1.8.190:39843 requested a full tablet report, sending...
I20260924 17:43:39.084636 7157 ts_manager.cc:194] Registered new tserver with Master: ea480d5a2def493c81aac3c1599ef1d1 (127.1.8.129:45267)
I20260924 17:43:39.085279 7157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:37039
WARNING: no policy specified for oryx/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab.
Entry for principal oryx/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab.
I20260924 17:43:39.109680 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39843
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:39.192445 7353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:39.192677 7353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:39.192701 7353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:39.193821 7353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:39.193859 7353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:39.193886 7353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:39.195322 7353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:39.195528 7353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:39.195719 7353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.200796 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.200807 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.200812 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.201069 7353 server_base.cc:1034] running on GCE node
I20260924 17:43:39.201294 7353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.201555 7353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.202714 7353 hybrid_clock.cc:648] HybridClock initialized: now 1790271819202704 us; error 32 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.205392 7353 init.cc:377] Logged in from keytab as oryx/127.1.8.130@KRBTEST.COM (short username oryx)
I20260924 17:43:39.205857 7353 webserver.cc:460] Webserver started at http://127.1.8.130:35233/ using document root <none> and password file <none>
I20260924 17:43:39.206081 7353 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.206148 7353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.206271 7353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:39.207360 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "c60e07ad0b9944a986191eab56f132af"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "acd9215b30e6f4c3f4657f4cd887217d"
server_key_iv: "589de1aca54405407badab9571a0f48d"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.207665 7353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "c60e07ad0b9944a986191eab56f132af"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "acd9215b30e6f4c3f4657f4cd887217d"
server_key_iv: "589de1aca54405407badab9571a0f48d"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.209472 7353 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 17:43:39.210158 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.210325 7353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:39.210435 7353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "c60e07ad0b9944a986191eab56f132af"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "acd9215b30e6f4c3f4657f4cd887217d"
server_key_iv: "589de1aca54405407badab9571a0f48d"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.210515 7353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.224341 7353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:39.226765 7353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:39.226923 7353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:39.227171 7353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:39.227522 7353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:39.227559 7353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.227599 7353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:39.227629 7353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.233560 7353 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:34319
I20260924 17:43:39.233634 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:34319 every 8 connection(s)
I20260924 17:43:39.233984 7353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:39.234794 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7353
I20260924 17:43:39.234910 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:43:39.235107 1058 external_mini_cluster.cc:1428] Setting key 86f30b711accdee9de4f5566f2ad0b57
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:39.245155 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.234786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:43835 (local address 127.1.8.190:39843)
0924 17:43:39.234882 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.234885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.235507 (+ 622us) server_negotiation.cc:407] Connection header received
0924 17:43:39.236151 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.236151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.236161 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.236201 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.237630 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.238135 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.239225 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.239533 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.239608 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.242121 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.242122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.242122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.242130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.243623 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.244156 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.244157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.244157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.244209 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.244489 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.244490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.244490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.244599 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.244649 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.245014 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.245087 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:43:39.245436 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.234764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:43835)
0924 17:43:39.235329 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.235389 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.236046 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.236281 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.236286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.237201 (+ 915us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.237510 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.237518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.238235 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.238236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.239135 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.239137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.239628 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.239629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.239754 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.240341 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.240351 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.242021 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.243714 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.243722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.243724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.244059 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.244272 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.244273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.244273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.244424 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.244665 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.244671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.244865 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.245006 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.245198 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:43:39.246007 7482 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39843
I20260924 17:43:39.246112 7482 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:39.246604 7482 heartbeater.cc:511] Master 127.1.8.190:39843 requested a full tablet report, sending...
I20260924 17:43:39.247027 7157 ts_manager.cc:194] Registered new tserver with Master: c60e07ad0b9944a986191eab56f132af (127.1.8.130:34319)
I20260924 17:43:39.247433 7157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:43835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab.
Entry for principal oryx/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab.
I20260924 17:43:39.281108 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39843
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:39.368199 7487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:39.368433 7487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:39.368458 7487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:39.369585 7487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:39.369623 7487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:39.369650 7487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:39.371132 7487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:39.371325 7487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:39.371527 7487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.376778 7495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.376778 7492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.376782 7493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.376863 7487 server_base.cc:1034] running on GCE node
I20260924 17:43:39.377171 7487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.377388 7487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.378541 7487 hybrid_clock.cc:648] HybridClock initialized: now 1790271819378522 us; error 30 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.381083 7487 init.cc:377] Logged in from keytab as oryx/127.1.8.131@KRBTEST.COM (short username oryx)
I20260924 17:43:39.381546 7487 webserver.cc:460] Webserver started at http://127.1.8.131:32775/ using document root <none> and password file <none>
I20260924 17:43:39.381767 7487 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.381829 7487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.381973 7487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:39.383097 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "05048dd9ae124416b36034608ad7f8d7"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "43caaa9531d29b2ed6070ed400bcc6a7"
server_key_iv: "036a36b5b439ece8aacdeacdeb809576"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.383453 7487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "05048dd9ae124416b36034608ad7f8d7"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "43caaa9531d29b2ed6070ed400bcc6a7"
server_key_iv: "036a36b5b439ece8aacdeacdeb809576"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.384792 7487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:39.385565 7502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.385754 7487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:39.385857 7487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "05048dd9ae124416b36034608ad7f8d7"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "43caaa9531d29b2ed6070ed400bcc6a7"
server_key_iv: "036a36b5b439ece8aacdeacdeb809576"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.385934 7487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.396023 7487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:39.398351 7487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:39.398536 7487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:39.398773 7487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:39.399158 7487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:39.399219 7487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.399263 7487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:39.399289 7487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.405460 7487 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:43985
I20260924 17:43:39.405530 7615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:43985 every 8 connection(s)
I20260924 17:43:39.405897 7487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:43:39.406157 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7487
I20260924 17:43:39.406260 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:43:39.406517 1058 external_mini_cluster.cc:1428] Setting key 69e080bf1bf8b104fc2d24fe2a96ec8d
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:39.416188 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.407004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:43445 (local address 127.1.8.190:39843)
0924 17:43:39.407052 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.407055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.407064 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:39.407731 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.407732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.407741 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.407787 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.408810 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.409371 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.410569 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.410774 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.410835 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.413050 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.413052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.413052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.413059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.414783 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.415190 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.415191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.415192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.415251 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.415583 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.415584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.415584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.415704 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.415765 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.416048 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.416117 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:43:39.416359 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.406705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:43445)
0924 17:43:39.406938 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.406989 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.407640 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.407843 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.407847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.408449 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.408682 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.408689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.409535 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.409536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.410464 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.410466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.410849 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.410850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.410970 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.411491 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.411499 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.412966 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.414863 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.414869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.414871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.415116 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.415339 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.415340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.415341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.415509 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.415767 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.415771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.415898 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.415997 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.416128 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260924 17:43:39.417117 7616 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39843
I20260924 17:43:39.417212 7616 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:39.417450 7616 heartbeater.cc:511] Master 127.1.8.190:39843 requested a full tablet report, sending...
I20260924 17:43:39.417898 7157 ts_manager.cc:194] Registered new tserver with Master: 05048dd9ae124416b36034608ad7f8d7 (127.1.8.131:43985)
I20260924 17:43:39.418268 7157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:43445
I20260924 17:43:39.418337 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.8.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.189.keytab.
Entry for principal oryx/127.1.8.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.189.keytab.
I20260924 17:43:39.446085 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:45075
--webserver_interface=127.1.8.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.189.keytab
--principal=oryx/127.1.8.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39843,127.1.8.189:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:39.529966 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:39.530222 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:39.530262 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:39.531548 7621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:39.531622 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:39.531648 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:39.531682 7621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:39.531708 7621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:39.533244 7621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39843,127.1.8.189:45075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.189.keytab
--principal=oryx/127.1.8.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.8.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
W20260924 17:43:39.533408 7621 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.8.190:39843,127.1.8.189:45075'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 17:43:39.533576 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:39.533818 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.539209 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.539222 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.539203 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.539240 7621 server_base.cc:1034] running on GCE node
I20260924 17:43:39.539716 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.540024 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.541164 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790271819541147 us; error 27 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.543860 7621 init.cc:377] Logged in from keytab as oryx/127.1.8.189@KRBTEST.COM (short username oryx)
I20260924 17:43:39.544425 7621 webserver.cc:460] Webserver started at http://127.1.8.189:34595/ using document root <none> and password file <none>
I20260924 17:43:39.544674 7621 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.544721 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.544888 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:39.545979 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/instance:
uuid: "2db65fc538c949deb183b4b171888aa3"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "9ca3160e7048468555cdd7e06ed5a315"
server_key_iv: "c3cd0a246e0470966d1535a3318bbde9"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.546342 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal/instance:
uuid: "2db65fc538c949deb183b4b171888aa3"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "9ca3160e7048468555cdd7e06ed5a315"
server_key_iv: "c3cd0a246e0470966d1535a3318bbde9"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.547749 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:39.548506 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.548692 7621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:39.548782 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
uuid: "2db65fc538c949deb183b4b171888aa3"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "9ca3160e7048468555cdd7e06ed5a315"
server_key_iv: "c3cd0a246e0470966d1535a3318bbde9"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.548856 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.562942 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:39.565179 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:39.565308 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:39.569603 7621 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.189:45075
I20260924 17:43:39.569665 7688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.189:45075 every 8 connection(s)
I20260924 17:43:39.570057 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
I20260924 17:43:39.571165 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7621
I20260924 17:43:39.571197 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:39.573618 7689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.189@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:39.581939 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.574285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.1.8.190:39843)
0924 17:43:39.574346 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.574349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.574641 (+ 292us) server_negotiation.cc:407] Connection header received
0924 17:43:39.574708 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.574710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.574725 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.574787 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.575236 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.575722 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.576719 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.576915 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.576987 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.579103 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.579105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.579106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.579113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.580591 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.580941 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.580942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.580942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.581002 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.581258 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.581259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.581259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.581400 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.581484 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.581745 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.581851 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:39.582077 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.571841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54146 (local address 127.1.8.189:45075)
0924 17:43:39.572009 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.572023 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.572068 (+ 45us) server_negotiation.cc:407] Connection header received
0924 17:43:39.572482 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.572489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.572538 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.572709 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.573381 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.574541 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.575290 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.575712 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.575828 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.578203 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.578208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.578209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.578214 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.580083 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.580512 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.580514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.580515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.580592 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.581117 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.581118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.581119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.581388 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.581534 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.581652 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.581822 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260924 17:43:39.582077 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.574221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:56694)
0924 17:43:39.574527 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.574569 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.574650 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.574815 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.574818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.575085 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.575150 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.575151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.575818 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.575819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.576627 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.576630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.576992 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.576993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.577071 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.577555 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.577563 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.579011 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.580665 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.580671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.580673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.580869 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.581071 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.581072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.581072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.581190 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.581495 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.581499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.581578 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.581694 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.581867 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":83,"thread_start_us":81,"threads_started":1}
I20260924 17:43:39.582839 7689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.189@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
I20260924 17:43:39.589659 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.583196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.1.8.189:45075)
0924 17:43:39.583232 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.583234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.583247 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:43:39.583305 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.583305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.583312 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.583353 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.583559 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.584015 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.584653 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.584867 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.584938 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.586794 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.586795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.586796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.586803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.588407 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.588776 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.588776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.588777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.588829 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.588999 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.589000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.589000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.589112 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.589183 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.589304 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.589430 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":19}
I20260924 17:43:39.589761 7693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.583144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.0.0.1:54158)
0924 17:43:39.583194 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.583206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.583247 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.583356 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.583357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.583467 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.583513 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.583515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.584082 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.584082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.584561 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.584562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.584939 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.584940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.584992 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.585369 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.585373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.586687 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.588513 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.588515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.588515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.588697 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.588891 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.588892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.588892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.588951 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.589174 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.589175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.589213 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.589278 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.589381 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:43:39.591163 7689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Bootstrap starting.
I20260924 17:43:39.591954 7689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:39.592185 7689 log.cc:824] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:39.592934 7689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: No bootstrap required, opened a new log
I20260924 17:43:39.594344 7689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } }
I20260924 17:43:39.594568 7689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:39.594599 7689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db65fc538c949deb183b4b171888aa3, State: Initialized, Role: FOLLOWER
I20260924 17:43:39.594723 7689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } }
I20260924 17:43:39.595019 7689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:39.594997 7695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } } }
I20260924 17:43:39.595167 7695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:43:39.596827 7706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:39.596892 7706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:39.598374 7621 master_runner.cc:428] Detected that this master 2db65fc538c949deb183b4b171888aa3 is joining an existing cluster
I20260924 17:43:39.598464 7621 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.8.189:45075
I20260924 17:43:39.606688 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.599640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.1.8.190:39843)
0924 17:43:39.599691 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.599693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.599886 (+ 193us) server_negotiation.cc:407] Connection header received
0924 17:43:39.599915 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.599916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.599930 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.599986 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.600480 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.601356 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.602485 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.602694 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.602782 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.603836 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.603837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.603837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.603844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.605378 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.605774 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.605775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.605776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.605832 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.606016 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.606017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.606017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.606134 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.606205 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.606504 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.606589 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 17:43:39.606719 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.599546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:56704)
0924 17:43:39.599703 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.599716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.599802 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.600036 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.600036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.600280 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.600381 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.600384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.601457 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.601458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.602258 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.602259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.602781 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.602783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.602843 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.603311 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.603317 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.603735 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.605475 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.605476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.605477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.605706 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.605900 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.605901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.605902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.605966 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.606196 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.606197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.606240 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.606355 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.606606 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260924 17:43:39.607411 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.599832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.1.8.189:45075)
0924 17:43:39.599868 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.599870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.599988 (+ 118us) server_negotiation.cc:407] Connection header received
0924 17:43:39.600017 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.600018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.600028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.600089 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.600818 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.601276 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.602345 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.602717 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.602795 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.604105 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.604106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.604106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.604113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.605877 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.606325 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.606326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.606327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.606379 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.606600 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.606601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.606602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.606706 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.606746 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.607243 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.607315 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:39.607800 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.599799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.0.0.1:54174)
0924 17:43:39.599911 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.599920 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.599977 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.600089 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.600090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.600681 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.600735 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.600736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.601350 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.601351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.602253 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.602256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.602849 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.602850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.602913 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.603497 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.603503 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.603983 (+ 480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.606035 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.606036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.606036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.606250 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.606471 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.606472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.606472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.606527 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.607110 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.607111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.607150 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.607623 (+ 473us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.607708 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.190@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
I20260924 17:43:39.616945 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.609592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54186 (local address 127.1.8.189:45075)
0924 17:43:39.609628 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.609630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.609786 (+ 156us) server_negotiation.cc:407] Connection header received
0924 17:43:39.609880 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.609881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.609892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.609947 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.610415 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.610914 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.611721 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.611913 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.611986 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.614315 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.614316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.614317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.614324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.615764 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.616159 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.616160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.616161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.616215 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.616407 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.616407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.616408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.616522 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.616590 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.616773 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.616848 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 17:43:39.616951 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.609532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.0.0.1:54186)
0924 17:43:39.609682 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.609727 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.609813 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.609953 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.609957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.610250 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.610301 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.610304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.611011 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.611013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.611627 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.611629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.611987 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.611988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.612065 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.612604 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.612610 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.614224 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.615863 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.615871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.615873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.616085 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.616283 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.616284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.616285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.616343 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.616583 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.616591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.616645 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.616735 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.616854 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260924 17:43:39.617534 7157 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.8.189:45075: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2db65fc538c949deb183b4b171888aa3"
member_type: NON_VOTER
last_known_addr {
host: "127.1.8.189"
port: 45075
}
attrs {
promote: true
}
}
dest_uuid: "d79e6ab0487943a99573b2a4e2c379d7"
cas_config_opid_index: -1
I20260924 17:43:39.617878 7157 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } }
I20260924 17:43:39.618439 7716 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2db65fc538c949deb183b4b171888aa3 (127.1.8.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } }
I20260924 17:43:39.618685 7663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:39.619460 7663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Refusing update from remote peer d79e6ab0487943a99573b2a4e2c379d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 17:43:39.619485 7716 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.8.189:45075
I20260924 17:43:39.619640 7716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } }
I20260924 17:43:39.619706 7716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:39.619729 7621 master.cc:537] Master@127.1.8.189:45075 shutting down...
I20260924 17:43:39.619808 7715 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:43:39.620491 7142 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 17:43:39.620986 7142 consensus_peers.cc:487] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 -> Peer 2db65fc538c949deb183b4b171888aa3 (127.1.8.189:45075): Couldn't send request to peer 2db65fc538c949deb183b4b171888aa3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 17:43:39.622866 7621 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:39.622973 7621 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:39.623026 7621 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: stopping tablet replica
I20260924 17:43:39.635656 7621 master.cc:552] Master@127.1.8.189:45075 shutdown complete.
I20260924 17:43:39.635741 7621 master_runner.cc:305] Clearing existing system tablet
I20260924 17:43:39.635886 7621 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.635958 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.636673 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:39.637197 7720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.637326 7621 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.637388 7621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.637421 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
uuid: "2db65fc538c949deb183b4b171888aa3"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "9ca3160e7048468555cdd7e06ed5a315"
server_key_iv: "c3cd0a246e0470966d1535a3318bbde9"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.638677 7621 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:39.640408 7621 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 17:43:39.640493 7621 log.cc:1196] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 17:43:39.640954 7621 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Deleting consensus metadata
I20260924 17:43:39.641120 7621 master_runner.cc:315] Copying system tablet from 127.1.8.190:39843
I20260924 17:43:39.641845 7621 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: tablet copy: Beginning tablet copy session from remote peer at address 127.1.8.190:39843
I20260924 17:43:39.648649 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.642299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.1.8.190:39843)
0924 17:43:39.642356 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.642359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.642540 (+ 181us) server_negotiation.cc:407] Connection header received
0924 17:43:39.642609 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.642610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.642620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.642672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.643075 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.643612 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.644713 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.644933 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.644995 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.646092 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.646093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.646093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.646100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.647560 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.647940 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.647941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.647941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.647991 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.648192 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.648192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.648193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.648306 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.648359 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.648521 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.648582 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:43:39.648711 7726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.642205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:56710)
0924 17:43:39.642367 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.642380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.642543 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.642690 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.642691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.642908 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.642993 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.642996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.643693 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.643694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.644619 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.644621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.645007 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.645009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.645081 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.645557 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.645562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.645984 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.647638 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.647639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.647640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.647869 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.648051 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.648052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.648052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.648117 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.648366 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.648367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.648416 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.648489 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.648597 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260924 17:43:39.649356 7177 tablet_copy_service.cc:140] P d79e6ab0487943a99573b2a4e2c379d7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2db65fc538c949deb183b4b171888aa3 ({username='oryx', principal='oryx/127.1.8.189@KRBTEST.COM'} at 127.0.0.1:56710)
I20260924 17:43:39.649443 7177 tablet_copy_service.cc:161] P d79e6ab0487943a99573b2a4e2c379d7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2db65fc538c949deb183b4b171888aa3 at {username='oryx', principal='oryx/127.1.8.189@KRBTEST.COM'} at 127.0.0.1:56710: session id = 2db65fc538c949deb183b4b171888aa3-00000000000000000000000000000000
I20260924 17:43:39.650130 7177 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 17:43:39.650789 7621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 17:43:39.651907 7621 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: tablet copy: Starting download of 0 data blocks...
I20260924 17:43:39.652120 7621 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: tablet copy: Starting download of 1 WAL segments...
I20260924 17:43:39.653077 7621 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 17:43:39.653855 7177 tablet_copy_service.cc:342] P d79e6ab0487943a99573b2a4e2c379d7: Request end of tablet copy session 2db65fc538c949deb183b4b171888aa3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.8.189@KRBTEST.COM'} at 127.0.0.1:56710
I20260924 17:43:39.653916 7177 tablet_copy_service.cc:433] P d79e6ab0487943a99573b2a4e2c379d7: ending tablet copy session 2db65fc538c949deb183b4b171888aa3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2db65fc538c949deb183b4b171888aa3
I20260924 17:43:39.656606 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.657965 7728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.657979 7730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.658058 7727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.658595 7621 server_base.cc:1034] running on GCE node
I20260924 17:43:39.658735 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.658917 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.660061 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790271819660052 us; error 27 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.661537 7621 init.cc:377] Logged in from keytab as oryx/127.1.8.189@KRBTEST.COM (short username oryx)
I20260924 17:43:39.661805 7621 webserver.cc:460] Webserver started at http://127.1.8.189:42701/ using document root <none> and password file <none>
I20260924 17:43:39.661924 7621 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.661970 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.662706 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:39.663257 7737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.663440 7621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:43:39.663491 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
uuid: "2db65fc538c949deb183b4b171888aa3"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "9ca3160e7048468555cdd7e06ed5a315"
server_key_iv: "c3cd0a246e0470966d1535a3318bbde9"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.663549 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.672263 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:39.673991 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:39.677230 7621 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.189:45075
I20260924 17:43:39.677686 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
I20260924 17:43:39.677696 7789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.189:45075 every 8 connection(s)
I20260924 17:43:39.678483 7790 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:43:39.678902 7790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Bootstrap starting.
I20260924 17:43:39.681118 7790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:43:39.681351 7790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: Bootstrap complete.
I20260924 17:43:39.681576 7790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } }
I20260924 17:43:39.681663 7790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2db65fc538c949deb183b4b171888aa3, State: Initialized, Role: LEARNER
I20260924 17:43:39.681712 7790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } }
I20260924 17:43:39.681888 7790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:39.681883 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } }
I20260924 17:43:39.681990 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: This master's current role is: LEARNER
I20260924 17:43:39.683408 7804 catalog_manager.cc:1259] Loaded cluster ID: 11715045a2fc44c08f8c2b623651aa38
I20260924 17:43:39.683478 7804 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: loading cluster ID for follower catalog manager: success
I20260924 17:43:39.684494 7804 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: acquiring CA information for follower catalog manager: success
I20260924 17:43:39.684772 7804 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:43:39.691459 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.684008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.1.8.189:45075)
0924 17:43:39.684884 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.684886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.684901 (+ 15us) server_negotiation.cc:407] Connection header received
0924 17:43:39.684922 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.684922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.684931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.685014 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.685537 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.686173 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.686972 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.687181 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.687256 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.688612 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.688614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.688614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.688622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.690217 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.690628 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.690629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.690629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.690691 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.690885 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.690886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.690886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.691007 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.691115 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.691253 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.691341 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for oryx/127.1.8.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.188.keytab.
Entry for principal oryx/127.1.8.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.188.keytab.
I20260924 17:43:39.721382 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:34117
--webserver_interface=127.1.8.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.188.keytab
--principal=oryx/127.1.8.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:43:39.818674 7808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:39.818888 7808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:39.818912 7808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:39.820134 7808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:43:39.820185 7808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:39.820200 7808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:39.820246 7808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:43:39.820271 7808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:43:39.821852 7808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.188.keytab
--principal=oryx/127.1.8.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.8.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:39.822124 7808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:39.822386 7808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.827219 7813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.827227 7814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.827310 7816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.827700 7808 server_base.cc:1034] running on GCE node
I20260924 17:43:39.827992 7808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.828297 7808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.829494 7808 hybrid_clock.cc:648] HybridClock initialized: now 1790271819829477 us; error 29 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.832105 7808 init.cc:377] Logged in from keytab as oryx/127.1.8.188@KRBTEST.COM (short username oryx)
I20260924 17:43:39.832624 7808 webserver.cc:460] Webserver started at http://127.1.8.188:46039/ using document root <none> and password file <none>
I20260924 17:43:39.832866 7808 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.832932 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.833081 7808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:43:39.834165 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/instance:
uuid: "5881523ba56c4e8aa460ae226e536ab9"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "aeb4e055a94e26fb168dac8d1d2498e9"
server_key_iv: "f2ee0c221effdfcd4179a56b446f90d6"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.834565 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal/instance:
uuid: "5881523ba56c4e8aa460ae226e536ab9"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "aeb4e055a94e26fb168dac8d1d2498e9"
server_key_iv: "f2ee0c221effdfcd4179a56b446f90d6"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.836035 7808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:39.836890 7823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.837066 7808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:43:39.837168 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
uuid: "5881523ba56c4e8aa460ae226e536ab9"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "aeb4e055a94e26fb168dac8d1d2498e9"
server_key_iv: "f2ee0c221effdfcd4179a56b446f90d6"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.837249 7808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.860291 7808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:39.862159 7808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:39.862330 7808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:39.867334 7808 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.188:34117
I20260924 17:43:39.867377 7875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.188:34117 every 8 connection(s)
I20260924 17:43:39.867758 7808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
I20260924 17:43:39.868824 7876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:39.870973 7876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.188@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:39.877274 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7808
I20260924 17:43:39.880038 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.871738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.1.8.190:39843)
0924 17:43:39.871812 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.871814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.871911 (+ 97us) server_negotiation.cc:407] Connection header received
0924 17:43:39.872369 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.872370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.872380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.872424 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.873047 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.873536 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.874601 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.874782 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.874838 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.876921 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.876923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.876923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.876929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.878582 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.879073 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.879075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.879075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.879143 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.879386 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.879386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.879387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.879526 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.879599 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.879850 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.879934 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 17:43:39.880228 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.871639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:56714)
0924 17:43:39.871795 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.871834 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.872274 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.872506 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.872509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.872765 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.872969 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.872973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.873622 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.873623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.874527 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.874529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.874851 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.874852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.874959 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.875437 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.875464 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.876827 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.878655 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.878663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.878666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.878991 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.879213 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.879213 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.879213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.879315 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.879593 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.879597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.879721 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.879810 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.879992 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":42,"thread_start_us":69,"threads_started":1}
I20260924 17:43:39.880730 7876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.8.188@KRBTEST.COM
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.188@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
I20260924 17:43:39.888127 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.877969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.1.8.188:34117)
0924 17:43:39.878134 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.878139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.878156 (+ 17us) server_negotiation.cc:407] Connection header received
0924 17:43:39.878174 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.878176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.878188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.878269 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.878995 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.880117 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.881174 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.881489 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.881567 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.884546 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.884555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.884555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.884561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.886746 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.887198 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.887207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.887207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.887265 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.887478 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.887479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.887480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.887646 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.887849 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.887896 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.887985 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260924 17:43:39.888296 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.881084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.1.8.189:45075)
0924 17:43:39.881128 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.881130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.881140 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:39.881177 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.881177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.881190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.881256 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.881525 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.882193 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.883015 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.883252 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.883320 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.885258 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.885260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.885260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.885269 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.887073 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.887499 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.887500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.887501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.887559 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.887747 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.887748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.887748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.887858 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.887945 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.888081 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.888208 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:39.888295 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.880992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.0.0.1:54212)
0924 17:43:39.881059 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.881071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.881120 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.881258 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.881259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.881398 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.881446 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.881447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.882339 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.882341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.882931 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.882933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.883329 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.883331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.883423 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.883915 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.883919 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.885167 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.887172 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.887173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.887174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.887339 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.887622 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.887623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.887623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.887679 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.887936 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.887937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.887978 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.888057 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.888160 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260924 17:43:39.888713 7876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 34117 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.188@KRBTEST.COM for oryx/127.1.8.188@KRBTEST.COM
I20260924 17:43:39.895582 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.889016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.0.0.1:50116)
0924 17:43:39.889068 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.889079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.889121 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.889237 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.889237 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.889392 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.889425 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.889426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.890144 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.890145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.890808 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.890810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.891177 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.891178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.891222 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.891611 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.891616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.892732 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.894524 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.894524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.894525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.894684 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.894889 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.894890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.894890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.894951 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.895270 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.895271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.895311 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.895421 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.895513 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:43:39.895733 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.889090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.1.8.188:34117)
0924 17:43:39.889129 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.889131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.889148 (+ 17us) server_negotiation.cc:407] Connection header received
0924 17:43:39.889164 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.889165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.889173 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.889220 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.889485 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.890069 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.890898 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.891090 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.891156 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.892808 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.892809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.892809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.892816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.894458 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.894740 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.894740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.894741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.894790 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.895048 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.895049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.895049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.895170 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.895265 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.895451 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.895584 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":24}
I20260924 17:43:39.897033 7876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Bootstrap starting.
I20260924 17:43:39.897873 7876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:39.898156 7876 log.cc:824] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:39.898936 7876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: No bootstrap required, opened a new log
I20260924 17:43:39.900425 7876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 34117 } }
I20260924 17:43:39.900560 7876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:39.900583 7876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5881523ba56c4e8aa460ae226e536ab9, State: Initialized, Role: FOLLOWER
I20260924 17:43:39.900729 7876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 34117 } }
I20260924 17:43:39.901064 7876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:39.901051 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 34117 } } }
I20260924 17:43:39.901160 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:39.901932 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:39.902033 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 34117 } }
I20260924 17:43:39.902282 7881 leader_election.cc:288] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843), 2db65fc538c949deb183b4b171888aa3 (127.1.8.189:45075)
I20260924 17:43:39.902673 7764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5881523ba56c4e8aa460ae226e536ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2db65fc538c949deb183b4b171888aa3" is_pre_election: true
I20260924 17:43:39.902670 7167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5881523ba56c4e8aa460ae226e536ab9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d79e6ab0487943a99573b2a4e2c379d7" is_pre_election: true
I20260924 17:43:39.902798 7764 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Handling vote request from an unknown peer 5881523ba56c4e8aa460ae226e536ab9
I20260924 17:43:39.902859 7764 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 5881523ba56c4e8aa460ae226e536ab9 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 17:43:39.903101 7825 leader_election.cc:302] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5881523ba56c4e8aa460ae226e536ab9; no voters: 2db65fc538c949deb183b4b171888aa3, d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:43:39.903276 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 0 FOLLOWER]: Advancing to term 1
W20260924 17:43:39.903411 7892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:43:39.903460 7892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:43:39.904088 7881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 17:43:39.905443 7808 master_runner.cc:428] Detected that this master 5881523ba56c4e8aa460ae226e536ab9 is joining an existing cluster
I20260924 17:43:39.905489 7808 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.8.188:34117
I20260924 17:43:39.914093 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.906839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.0.0.1:54224)
0924 17:43:39.907002 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.907013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.907089 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.907291 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.907291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.907514 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.907588 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.907590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.908539 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.908540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.909472 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.909474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.909986 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.910052 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.910631 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.910636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.911002 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.912770 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.912771 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.912771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.912976 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.913184 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.913185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.913185 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.913231 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.913529 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.913530 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.913845 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.913924 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.914011 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":46,"thread_start_us":65,"threads_started":1}
I20260924 17:43:39.914158 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.906903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.1.8.189:45075)
0924 17:43:39.906946 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.906948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.907064 (+ 116us) server_negotiation.cc:407] Connection header received
0924 17:43:39.907166 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.907167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.907186 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.907221 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.907680 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.908442 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909569 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.909866 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909912 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.911087 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.911088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.911089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.911096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.912674 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.913053 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.913054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.913054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.913118 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.913285 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.913286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.913286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.913459 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.913531 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.913975 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.914063 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:39.930959 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.906671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56720 (local address 127.1.8.190:39843)
0924 17:43:39.906716 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.906718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.906844 (+ 126us) server_negotiation.cc:407] Connection header received
0924 17:43:39.906948 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.906949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.906960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.906991 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.907462 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.908249 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909757 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.910048 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.910090 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.927793 (+ 17703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.927796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.927796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.927805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.929628 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.930090 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.930093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.930093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.930158 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.930333 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.930334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.930334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.930526 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.930596 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.930779 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.930861 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:39.930948 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.907125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.1.8.188:34117)
0924 17:43:39.907157 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.907158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.907260 (+ 102us) server_negotiation.cc:407] Connection header received
0924 17:43:39.907337 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.907338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.907346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.907385 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.907716 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.908445 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909577 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.909851 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909888 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.928169 (+ 18281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.928170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.928170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.928176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.929765 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.930160 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.930162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.930162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.930211 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.930371 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.930372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.930372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.930528 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.930605 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.930745 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.930832 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:43:39.931034 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.907000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.0.0.1:50130)
0924 17:43:39.907214 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.907222 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.907279 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.907434 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.907435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.907598 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.907646 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.907647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.908554 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.908555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.909472 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.909474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.909985 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.909986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.910052 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.927611 (+ 17559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.927617 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.928081 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.929851 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.929852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.929852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.930089 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.930268 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.930269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.930269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.930322 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.930596 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.930597 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.930640 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.930726 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.930885 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":103,"thread_start_us":66,"threads_started":1}
I20260924 17:43:39.931300 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.906624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:56720)
0924 17:43:39.906775 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.906787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.906873 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.907057 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.907058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.907278 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.907364 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.907366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.908362 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.908363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.909651 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.909654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.910168 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.910169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.910231 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.927157 (+ 16926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.927165 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.927660 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.929733 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.929735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.929735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.930005 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.930218 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.930218 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.930219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.930277 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.930624 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.930625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.930668 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.931087 (+ 419us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.931192 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271818, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.190@KRBTEST.COM for oryx/127.1.8.188@KRBTEST.COM
I20260924 17:43:39.939710 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7219
I20260924 17:43:39.940726 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.933751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.0.0.1:50146)
0924 17:43:39.933803 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.933813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.933850 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.933978 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.933979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.934141 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.934187 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.934189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.934869 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.934870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.935391 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.935393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.935763 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.935764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.935839 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.936266 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.936271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.937633 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.939544 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.939545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.939546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.939770 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.940010 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.940011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.940011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.940070 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.940381 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.940382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.940438 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.940499 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.940622 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:43:39.941108 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.933816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.1.8.188:34117)
0924 17:43:39.933869 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.933870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.933878 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:39.933915 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.933916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.933926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.933976 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.934266 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.934795 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.935475 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.935681 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.935763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.937750 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.937751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.937752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.937758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.939395 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.939855 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.939856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.939857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.939942 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.940131 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.940132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.940132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.940289 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.940345 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.940915 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.941019 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:43:39.945729 7157 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.8.188:34117: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9"
member_type: NON_VOTER
last_known_addr {
host: "127.1.8.188"
port: 34117
}
attrs {
promote: true
}
}
dest_uuid: "d79e6ab0487943a99573b2a4e2c379d7"
cas_config_opid_index: 5
I20260924 17:43:39.945875 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:45267
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:39843
--webserver_port=45941
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:39.946043 7157 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:39.946506 7715 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5881523ba56c4e8aa460ae226e536ab9 (127.1.8.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.947533 7850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 1 FOLLOWER]: Refusing update from remote peer d79e6ab0487943a99573b2a4e2c379d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 17:43:39.947595 7715 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.8.188:34117
I20260924 17:43:39.947669 7715 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.947742 7715 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:39.947919 7808 master.cc:537] Master@127.1.8.188:34117 shutting down...
I20260924 17:43:39.947953 7716 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:43:39.949668 7142 consensus_peers.cc:487] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 -> Peer 5881523ba56c4e8aa460ae226e536ab9 (127.1.8.188:34117): Couldn't send request to peer 5881523ba56c4e8aa460ae226e536ab9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 17:43:39.951779 7808 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:39.951889 7808 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:39.951949 7808 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: stopping tablet replica
I20260924 17:43:39.954773 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.946606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.0.0.1:54232)
0924 17:43:39.946736 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.946762 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.946802 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.949883 (+ 3081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.949886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.949888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:39.949957 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.949959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.951064 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.951066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.952972 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.952978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.954252 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.954253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.954492 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.954533 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.954595 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.954660 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 17:43:39.955233 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.947905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.1.8.189:45075)
0924 17:43:39.948385 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.948389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.948401 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:43:39.948431 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.948432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.948459 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.948538 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:39.950056 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.950923 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.953103 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.953849 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.953908 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.953999 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.955082 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.955142 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260924 17:43:39.955246 7808 master.cc:552] Master@127.1.8.188:34117 shutdown complete.
I20260924 17:43:39.955286 7808 master_runner.cc:305] Clearing existing system tablet
I20260924 17:43:39.955420 7808 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.955470 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.955801 7764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Refusing update from remote peer d79e6ab0487943a99573b2a4e2c379d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 17:43:39.956127 7715 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 17:43:39.956846 7764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5881523ba56c4e8aa460ae226e536ab9 (127.1.8.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.958636 7808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:39.958817 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d79e6ab0487943a99573b2a4e2c379d7. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.958901 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: This master's current role is: LEARNER
I20260924 17:43:39.958947 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: NON_VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: true } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.958988 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: This master's current role is: LEARNER
I20260924 17:43:39.959102 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.959197 7808 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:39.959235 7808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.959256 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
uuid: "5881523ba56c4e8aa460ae226e536ab9"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "aeb4e055a94e26fb168dac8d1d2498e9"
server_key_iv: "f2ee0c221effdfcd4179a56b446f90d6"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.959452 7715 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: attempting to promote NON_VOTER 2db65fc538c949deb183b4b171888aa3 to VOTER
I20260924 17:43:39.959676 7715 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:39.960057 7808 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:39.960536 7764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 LEARNER]: Refusing update from remote peer d79e6ab0487943a99573b2a4e2c379d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 17:43:39.960819 7716 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 17:43:39.961320 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.960311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.0.0.1:50162)
0924 17:43:39.961233 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.961285 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":67,"threads_started":1}
I20260924 17:43:39.961648 7808 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 17:43:39.961731 7808 log.cc:1196] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 17:43:39.961709 7716 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2db65fc538c949deb183b4b171888aa3 (127.1.8.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.961971 7808 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Deleting consensus metadata
I20260924 17:43:39.962097 7808 master_runner.cc:315] Copying system tablet from 127.1.8.190:39843
I20260924 17:43:39.962651 7716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.962742 7716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:39.962801 7716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.962853 7716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 [sys.catalog]: This master's current role is: LEADER
I20260924 17:43:39.962836 7808 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: tablet copy: Beginning tablet copy session from remote peer at address 127.1.8.190:39843
I20260924 17:43:39.963130 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.961604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.0.0.1:50164)
0924 17:43:39.963052 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.963087 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:34117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1433}
I20260924 17:43:39.963148 7764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2db65fc538c949deb183b4b171888aa3 (127.1.8.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
W20260924 17:43:39.963702 7142 consensus_peers.cc:487] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7 -> Peer 5881523ba56c4e8aa460ae226e536ab9 (127.1.8.188:34117): Couldn't send request to peer 5881523ba56c4e8aa460ae226e536ab9. Status: Network error: Client connection negotiation failed: client connection to 127.1.8.188:34117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 17:43:39.964507 7793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d79e6ab0487943a99573b2a4e2c379d7" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:39.964598 7793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:43:39.970139 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.963651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.1.8.190:39843)
0924 17:43:39.963702 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:39.963704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:39.963860 (+ 156us) server_negotiation.cc:407] Connection header received
0924 17:43:39.963959 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:39.963960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:39.963970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:39.964001 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:39.964474 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.965130 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.965854 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.966082 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.966193 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.967471 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:39.967473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:39.967474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:39.967482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:39.968933 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.969319 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.969320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.969321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.969383 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:39.969604 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:39.969605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:39.969605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:39.969723 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:39.969792 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:39.969950 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 17:43:39.970032 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:43:39.970176 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:39.963515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:56736)
0924 17:43:39.963795 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:39.963806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:39.963886 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:39.964080 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:39.964081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:39.964310 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:39.964370 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.964372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.965216 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.965217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.965766 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:39.965768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:39.966196 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:39.966197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:39.966256 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:39.967007 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:39.967018 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:39.967351 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:39.969039 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.969040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.969040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.969235 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.969448 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:39.969449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:39.969450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:39.969526 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:39.969793 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:39.969794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:39.969842 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 17:43:39.969922 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:43:39.970033 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":71,"threads_started":1}
I20260924 17:43:39.970602 7177 tablet_copy_service.cc:140] P d79e6ab0487943a99573b2a4e2c379d7: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5881523ba56c4e8aa460ae226e536ab9 ({username='oryx', principal='oryx/127.1.8.188@KRBTEST.COM'} at 127.0.0.1:56736)
I20260924 17:43:39.970687 7177 tablet_copy_service.cc:161] P d79e6ab0487943a99573b2a4e2c379d7: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5881523ba56c4e8aa460ae226e536ab9 at {username='oryx', principal='oryx/127.1.8.188@KRBTEST.COM'} at 127.0.0.1:56736: session id = 5881523ba56c4e8aa460ae226e536ab9-00000000000000000000000000000000
I20260924 17:43:39.971163 7177 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 17:43:39.971681 7808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 17:43:39.972715 7808 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: tablet copy: Starting download of 0 data blocks...
I20260924 17:43:39.972885 7808 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: tablet copy: Starting download of 1 WAL segments...
I20260924 17:43:39.973614 7808 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 17:43:39.974545 7177 tablet_copy_service.cc:342] P d79e6ab0487943a99573b2a4e2c379d7: Request end of tablet copy session 5881523ba56c4e8aa460ae226e536ab9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.8.188@KRBTEST.COM'} at 127.0.0.1:56736
I20260924 17:43:39.974617 7177 tablet_copy_service.cc:433] P d79e6ab0487943a99573b2a4e2c379d7: ending tablet copy session 5881523ba56c4e8aa460ae226e536ab9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5881523ba56c4e8aa460ae226e536ab9
I20260924 17:43:39.977007 7808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:39.978253 7914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.978524 7915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:39.978932 7917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:39.985083 7808 server_base.cc:1034] running on GCE node
I20260924 17:43:39.985248 7808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:39.985528 7808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:39.986670 7808 hybrid_clock.cc:648] HybridClock initialized: now 1790271819986664 us; error 33 us; skew 500 ppm
Sep 24 17:43:39 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271819, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:39.988270 7808 init.cc:377] Logged in from keytab as oryx/127.1.8.188@KRBTEST.COM (short username oryx)
I20260924 17:43:39.988641 7808 webserver.cc:460] Webserver started at http://127.1.8.188:43453/ using document root <none> and password file <none>
I20260924 17:43:39.988780 7808 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:39.988819 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:39.989626 7808 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:39.990255 7924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:39.990450 7808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:43:39.990500 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
uuid: "5881523ba56c4e8aa460ae226e536ab9"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "aeb4e055a94e26fb168dac8d1d2498e9"
server_key_iv: "f2ee0c221effdfcd4179a56b446f90d6"
server_key_version: "encryptionkey@0"
I20260924 17:43:39.990572 7808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:40.000906 7808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:40.002894 7808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:40.006381 7808 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.188:34117
I20260924 17:43:40.006419 7976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.188:34117 every 8 connection(s)
I20260924 17:43:40.006807 7808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
I20260924 17:43:40.007604 7977 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:43:40.008068 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Bootstrap starting.
I20260924 17:43:40.010723 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:43:40.011003 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: Bootstrap complete.
I20260924 17:43:40.011262 7977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:40.011369 7977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5881523ba56c4e8aa460ae226e536ab9, State: Initialized, Role: LEARNER
I20260924 17:43:40.011421 7977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:40.011653 7977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:43:40.011642 7980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } } }
I20260924 17:43:40.011878 7980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9 [sys.catalog]: This master's current role is: LEARNER
I20260924 17:43:40.013391 7991 catalog_manager.cc:1259] Loaded cluster ID: 11715045a2fc44c08f8c2b623651aa38
I20260924 17:43:40.013926 7991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: loading cluster ID for follower catalog manager: success
I20260924 17:43:40.014922 7991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: acquiring CA information for follower catalog manager: success
I20260924 17:43:40.015244 7991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5881523ba56c4e8aa460ae226e536ab9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 17:43:40.069294 7901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:40.069522 7901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:40.069548 7901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:40.070807 7901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:40.070852 7901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:40.070886 7901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:43:40.072392 7901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:45267
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=45941
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:40.072661 7901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:40.072914 7901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:40.078177 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:40.078225 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:40.078293 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:40.078625 7901 server_base.cc:1034] running on GCE node
I20260924 17:43:40.078868 7901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:40.079124 7901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:40.080305 7901 hybrid_clock.cc:648] HybridClock initialized: now 1790271820080289 us; error 33 us; skew 500 ppm
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:40.083248 7901 init.cc:377] Logged in from keytab as oryx/127.1.8.129@KRBTEST.COM (short username oryx)
I20260924 17:43:40.083714 7901 webserver.cc:460] Webserver started at http://127.1.8.129:45941/ using document root <none> and password file <none>
I20260924 17:43:40.083913 7901 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:40.083967 7901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:40.085350 7901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:40.085915 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.086126 7901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.086227 7901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "ea480d5a2def493c81aac3c1599ef1d1"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "cb48ac273b5881ceb2321fead4486637"
server_key_iv: "feea395cf2e3dd7b0a4da2ca29c13b02"
server_key_version: "encryptionkey@0"
I20260924 17:43:40.086645 7901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:40.117653 7901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:40.119948 7901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:40.120127 7901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:40.120378 7901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:40.120700 7901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:40.120782 7901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.120843 7901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:40.120870 7901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.127553 7901 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:45267
I20260924 17:43:40.127610 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:45267 every 8 connection(s)
I20260924 17:43:40.128022 7901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:43:40.133255 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 7901
I20260924 17:43:40.133440 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7353
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.188@KRBTEST.COM
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:40.140918 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.129409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49931 (local address 127.1.8.188:34117)
0924 17:43:40.129504 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.129506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.129516 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:40.130453 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.130455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.130468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.130532 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.131330 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.131993 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.133024 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.133273 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.133389 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.137140 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.137141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.137142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.137150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.139078 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.139597 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.139598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.139599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.139657 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.139924 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.139925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.139925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.140086 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.140147 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.140667 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.140766 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
I20260924 17:43:40.140944 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.129226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:60137 (local address 127.1.8.189:45075)
0924 17:43:40.129273 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.129276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.129382 (+ 106us) server_negotiation.cc:407] Connection header received
0924 17:43:40.130435 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.130437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.130448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.130492 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.131344 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.132107 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.133247 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.133542 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.133600 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.137098 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.137100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.137100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.137109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.139583 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.139980 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.139981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.139981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.140037 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.140251 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.140252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.140252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.140394 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.140552 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.140784 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.140856 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:40.141127 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.129151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.1.8.129:60137)
0924 17:43:40.129320 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.129330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.130224 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.130621 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.130623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.131079 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.131235 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.131240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.132215 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.132215 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.133159 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.133162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.134033 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.134034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.134086 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.134449 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.134455 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.136985 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.139680 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.139681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.139681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.139898 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.140123 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.140124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.140124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.140185 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.140577 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.140578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.140623 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.140689 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.140855 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":112,"thread_start_us":81,"threads_started":1}
I20260924 17:43:40.141248 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:34319
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:39843
--webserver_port=35233
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:40.141506 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.129328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.1.8.129:49931)
0924 17:43:40.129446 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.129454 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.130236 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.130626 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.130627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.131030 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.131234 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.131238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.132112 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.132113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.132933 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.132934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.133357 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.133358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.133518 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.134269 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.134280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.137031 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.139171 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.139179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.139181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.139516 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.139720 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.139720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.139721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.139860 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.140173 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.140180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.140368 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.140482 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.141380 (+ 898us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":54,"threads_started":1}
I20260924 17:43:40.142134 8120 heartbeater.cc:347] Connected to a master server at 127.1.8.188:34117
I20260924 17:43:40.142246 8120 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.142457 8122 heartbeater.cc:347] Connected to a master server at 127.1.8.189:45075
I20260924 17:43:40.142474 8120 heartbeater.cc:511] Master 127.1.8.188:34117 requested a full tablet report, sending...
I20260924 17:43:40.142553 8122 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.142622 8122 heartbeater.cc:511] Master 127.1.8.189:45075 requested a full tablet report, sending...
I20260924 17:43:40.143087 7941 ts_manager.cc:194] Registered new tserver with Master: ea480d5a2def493c81aac3c1599ef1d1 (127.1.8.129:45267)
I20260924 17:43:40.143077 7754 ts_manager.cc:194] Registered new tserver with Master: ea480d5a2def493c81aac3c1599ef1d1 (127.1.8.129:45267)
I20260924 17:43:40.143128 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.128864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:55325 (local address 127.1.8.190:39843)
0924 17:43:40.128911 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.128912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.129178 (+ 266us) server_negotiation.cc:407] Connection header received
0924 17:43:40.130344 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.130345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.130358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.130555 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.131348 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.132102 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.135543 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.135775 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.135839 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.138574 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.138576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.138576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.138584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.140364 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.141782 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.141783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.141783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.141849 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.142308 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.142310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.142310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.142625 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.142679 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.142911 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.143008 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:43:40.143710 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.128872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:55325)
0924 17:43:40.129046 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.129085 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.130221 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.130618 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.130623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.131061 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.131235 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.131240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.132213 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.132214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.135449 (+ 3235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.135451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.135896 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.135897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.135966 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.136436 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.136442 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.138455 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.140462 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.140463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.140463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.140694 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.142110 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.142111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.142111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.142179 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.142733 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.142734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.142783 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.143208 (+ 425us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.143323 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":28,"thread_start_us":89,"threads_started":1}
I20260924 17:43:40.144071 8121 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39843
I20260924 17:43:40.144131 8121 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.144193 8121 heartbeater.cc:511] Master 127.1.8.190:39843 requested a full tablet report, sending...
I20260924 17:43:40.144439 7157 ts_manager.cc:194] Re-registered known tserver with Master: ea480d5a2def493c81aac3c1599ef1d1 (127.1.8.129:45267)
I20260924 17:43:40.144781 7157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:55325
W20260924 17:43:40.228709 8128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:40.228964 8128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:40.229013 8128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:40.230232 8128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:40.230279 8128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:40.230306 8128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:43:40.231848 8128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:34319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=35233
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:40.232057 8128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:40.232257 8128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:40.237593 8134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:40.237558 8136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:40.237681 8128 server_base.cc:1034] running on GCE node
W20260924 17:43:40.237649 8133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:40.238086 8128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:40.238350 8128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:40.239533 8128 hybrid_clock.cc:648] HybridClock initialized: now 1790271820239506 us; error 42 us; skew 500 ppm
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:40.242349 8128 init.cc:377] Logged in from keytab as oryx/127.1.8.130@KRBTEST.COM (short username oryx)
I20260924 17:43:40.242864 8128 webserver.cc:460] Webserver started at http://127.1.8.130:35233/ using document root <none> and password file <none>
I20260924 17:43:40.243100 8128 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:40.243184 8128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:40.244678 8128 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:40.245337 8143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.245550 8128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:40.245647 8128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "c60e07ad0b9944a986191eab56f132af"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "acd9215b30e6f4c3f4657f4cd887217d"
server_key_iv: "589de1aca54405407badab9571a0f48d"
server_key_version: "encryptionkey@0"
I20260924 17:43:40.246031 8128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:40.262817 8128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:40.265172 8128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:40.265328 8128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:40.265578 8128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:40.265899 8128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:40.265951 8128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.266000 8128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:40.266026 8128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.271957 8128 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:34319
I20260924 17:43:40.271998 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:34319 every 8 connection(s)
I20260924 17:43:40.272367 8128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:43:40.278708 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 8128
I20260924 17:43:40.278890 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7487
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.188@KRBTEST.COM
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
I20260924 17:43:40.286239 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:43985
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:39843
--webserver_port=32775
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:36059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:43:40.288551 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.273643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:42567 (local address 127.1.8.188:34117)
0924 17:43:40.273689 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.273691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.273751 (+ 60us) server_negotiation.cc:407] Connection header received
0924 17:43:40.274552 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.274553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.274571 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.274636 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.275613 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.276257 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277192 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.277459 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277527 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.280829 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.280830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.280831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.280839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.284604 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.285387 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.285388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.285389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.285475 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.286083 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.286085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.286085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.286596 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.286713 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.288332 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.288446 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:40.288758 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.273485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:36557 (local address 127.1.8.190:39843)
0924 17:43:40.273528 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.273529 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.273619 (+ 90us) server_negotiation.cc:407] Connection header received
0924 17:43:40.274155 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.274157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.274171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.274222 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.275285 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.275985 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277340 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.277623 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277696 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.280765 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.280767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.280767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.280781 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.282913 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.287855 (+ 4942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.287857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.287857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.287929 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.288154 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.288155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.288155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.288291 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.288350 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.288573 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.288664 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:40.289980 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.273155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:46109 (local address 127.1.8.189:45075)
0924 17:43:40.273211 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.273213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.273423 (+ 210us) server_negotiation.cc:407] Connection header received
0924 17:43:40.274281 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.274282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.274293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.274355 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.275631 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.276185 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277366 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.277636 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277698 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.286590 (+ 8892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.286592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.286593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.286603 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.288579 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.289148 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.289150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.289150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.289209 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.289405 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.289405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.289406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.289543 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.289588 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.289802 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.289880 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:43:40.289979 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.273416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:36557)
0924 17:43:40.273519 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.273528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.274047 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.274722 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.274723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.274948 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.275188 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.275193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.276101 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.276102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.277255 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.277258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.277695 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.277767 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.278501 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.278505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.280655 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.287488 (+ 6833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.287489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.287490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.287742 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.288008 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.288009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.288010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.288073 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.288386 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.288388 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.288447 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.289232 (+ 785us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.289572 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":27,"thread_start_us":59,"threads_started":1}
I20260924 17:43:40.290318 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.273151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.1.8.130:46109)
0924 17:43:40.273287 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.273324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.274029 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.274409 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.274415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.274957 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.275517 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.275519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.276276 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.276277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.277255 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.277258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.277752 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.277814 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.278432 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.278436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.286475 (+ 8039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.288814 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.288815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.288815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.289059 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.289281 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.289282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.289282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.289334 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.289636 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.289637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.289677 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.290100 (+ 423us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.290208 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":31,"thread_start_us":75,"threads_started":1}
I20260924 17:43:40.291381 8258 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39843
I20260924 17:43:40.291504 8258 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.291731 8258 heartbeater.cc:511] Master 127.1.8.190:39843 requested a full tablet report, sending...
I20260924 17:43:40.292173 8259 heartbeater.cc:347] Connected to a master server at 127.1.8.189:45075
I20260924 17:43:40.292238 7157 ts_manager.cc:194] Re-registered known tserver with Master: c60e07ad0b9944a986191eab56f132af (127.1.8.130:34319)
I20260924 17:43:40.292279 8259 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.292559 8259 heartbeater.cc:511] Master 127.1.8.189:45075 requested a full tablet report, sending...
I20260924 17:43:40.292557 8263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.273571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.1.8.130:42567)
0924 17:43:40.273675 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.273686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.274029 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.274663 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.274663 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.274950 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.275517 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.275519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.276329 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.276330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.277099 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.277101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.277540 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.277541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.277677 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.278384 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.278395 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.280690 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.284723 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.284733 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.284737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.285269 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.285601 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.285602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.285602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.285898 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.286800 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.286807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.288101 (+ 1294us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.291988 (+ 3887us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.292140 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260924 17:43:40.292726 7157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:36557
I20260924 17:43:40.293113 8257 heartbeater.cc:347] Connected to a master server at 127.1.8.188:34117
I20260924 17:43:40.293164 8257 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.293236 8257 heartbeater.cc:511] Master 127.1.8.188:34117 requested a full tablet report, sending...
I20260924 17:43:40.293525 7941 ts_manager.cc:194] Registered new tserver with Master: c60e07ad0b9944a986191eab56f132af (127.1.8.130:34319)
I20260924 17:43:40.293905 7754 ts_manager.cc:194] Registered new tserver with Master: c60e07ad0b9944a986191eab56f132af (127.1.8.130:34319)
W20260924 17:43:40.373903 8264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:43:40.374117 8264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:43:40.374142 8264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:43:40.375380 8264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:43:40.375433 8264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:43:40.375463 8264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:43:40.376966 8264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:36059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:43985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=32775
--tserver_master_addrs=127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:43:40.377225 8264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:43:40.377480 8264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:43:40.382567 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:43:40.382721 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:40.382820 8264 server_base.cc:1034] running on GCE node
W20260924 17:43:40.382908 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:43:40.383116 8264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:43:40.383399 8264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:43:40.384567 8264 hybrid_clock.cc:648] HybridClock initialized: now 1790271820384556 us; error 28 us; skew 500 ppm
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:43:40.387310 8264 init.cc:377] Logged in from keytab as oryx/127.1.8.131@KRBTEST.COM (short username oryx)
I20260924 17:43:40.387778 8264 webserver.cc:460] Webserver started at http://127.1.8.131:32775/ using document root <none> and password file <none>
I20260924 17:43:40.388000 8264 fs_manager.cc:359] Metadata directory not provided
I20260924 17:43:40.388060 8264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:43:40.389910 8264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:43:40.390671 8279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.390928 8264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:43:40.391037 8264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "05048dd9ae124416b36034608ad7f8d7"
format_stamp: "Formatted at 2026-09-24 17:43:39 on dist-test-slave-r7hf"
server_key: "43caaa9531d29b2ed6070ed400bcc6a7"
server_key_iv: "036a36b5b439ece8aacdeacdeb809576"
server_key_version: "encryptionkey@0"
I20260924 17:43:40.391459 8264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:43:40.409816 8264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:43:40.412809 8264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:43:40.413000 8264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:43:40.413290 8264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:43:40.413625 8264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:43:40.413661 8264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.413705 8264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:43:40.413734 8264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:43:40.420053 8264 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:43985
I20260924 17:43:40.420125 8392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:43985 every 8 connection(s)
I20260924 17:43:40.420522 8264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:43:40.422765 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 8264
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.188@KRBTEST.COM
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.189@KRBTEST.COM
Sep 24 17:43:40 dist-test-slave-r7hf krb5kdc[7112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271820, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:43:40.434092 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.421775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:49349 (local address 127.1.8.189:45075)
0924 17:43:40.421825 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.421827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.421898 (+ 71us) server_negotiation.cc:407] Connection header received
0924 17:43:40.422526 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.422528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.422540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.422586 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.423518 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.424271 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425270 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.425588 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425648 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.430232 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.430234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.430234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.430242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.432173 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.432730 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.432731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.432732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.432808 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.433185 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.433186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.433187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.433334 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.433388 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.433877 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.433976 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:43:40.435091 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.421705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45075 (local address 127.1.8.131:49349)
0924 17:43:40.421824 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.421834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.422340 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.422693 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.422699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.423100 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.423398 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.423405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.424360 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.424361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.425205 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.425207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.425698 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.425873 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.426623 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.426635 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.430125 (+ 3490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.432259 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.432267 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.432270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.432630 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.432894 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.432894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.432895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.433090 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.433463 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.433470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.433683 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.434556 (+ 873us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.434798 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260924 17:43:40.435691 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.421338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:34117 (local address 127.1.8.131:41729)
0924 17:43:40.421597 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.421646 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.422339 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.423672 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.423673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.423822 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.423874 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.423876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.424885 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.424886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.425551 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.425553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.426069 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.426070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.426134 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.426786 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.426791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.429530 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.434512 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.434513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.434514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.434739 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.434972 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.434973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.434973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.435045 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.435329 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.435330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.435373 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.435455 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.435557 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":145,"threads_started":1}
I20260924 17:43:40.435921 8395 heartbeater.cc:347] Connected to a master server at 127.1.8.189:45075
I20260924 17:43:40.436023 8395 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.436255 8395 heartbeater.cc:511] Master 127.1.8.189:45075 requested a full tablet report, sending...
I20260924 17:43:40.436751 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.421315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:41729 (local address 127.1.8.188:34117)
0924 17:43:40.421385 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.421387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.421736 (+ 349us) server_negotiation.cc:407] Connection header received
0924 17:43:40.422486 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.422487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.422501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.422542 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.423962 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.424699 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425636 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.425936 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.426000 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.429628 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.429629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.429630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.429637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.434424 (+ 4787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.434834 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.434835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.434835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.434902 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.435113 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.435114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.435114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.435222 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.435303 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.436577 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.436657 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:43:40.436779 7754 ts_manager.cc:194] Registered new tserver with Master: 05048dd9ae124416b36034608ad7f8d7 (127.1.8.131:43985)
I20260924 17:43:40.437065 8393 heartbeater.cc:347] Connected to a master server at 127.1.8.188:34117
I20260924 17:43:40.437112 8393 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.437186 8393 heartbeater.cc:511] Master 127.1.8.188:34117 requested a full tablet report, sending...
I20260924 17:43:40.437732 7941 ts_manager.cc:194] Registered new tserver with Master: 05048dd9ae124416b36034608ad7f8d7 (127.1.8.131:43985)
I20260924 17:43:40.437739 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.426438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.1.8.188:34117)
0924 17:43:40.426666 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.426668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.427971 (+ 1303us) server_negotiation.cc:407] Connection header received
0924 17:43:40.428005 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.428006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.428019 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.428093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.428988 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.429767 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.432086 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.432428 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.432498 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.433955 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.433957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.433957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.433964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.436283 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.436746 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.436747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.436747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.436838 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.437083 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.437084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.437084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.437234 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.437330 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.437528 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.437611 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":67,"threads_started":1}
I20260924 17:43:40.438629 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.421921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:59305)
0924 17:43:40.422053 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.422062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.422381 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.423057 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.423058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.423219 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:43:40.423406 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.423408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.424185 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.424186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.425128 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.425130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.425697 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.425873 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.426947 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:43:40.426952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:43:40.430802 (+ 3850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:43:40.437379 (+ 6577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.437380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.437381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.437603 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.437855 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:43:40.437856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:43:40.437856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:43:40.437919 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:43:40.438240 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:43:40.438241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:43:40.438295 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.438365 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.438493 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":31,"thread_start_us":60,"threads_started":1}
I20260924 17:43:40.438746 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.422091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:59305 (local address 127.1.8.190:39843)
0924 17:43:40.422132 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.422134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.422142 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:40.422501 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.422502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.422515 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.422569 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.423499 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.424059 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425216 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.425550 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.425621 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.430914 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.430917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.430917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.430927 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.434109 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.437696 (+ 3587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.437698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.437698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.437762 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.437995 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.437996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.437996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.438144 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.438202 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.438487 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.438639 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:43:40.439095 8394 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39843
I20260924 17:43:40.439141 8394 heartbeater.cc:464] Registering TS with master...
I20260924 17:43:40.439208 8394 heartbeater.cc:511] Master 127.1.8.190:39843 requested a full tablet report, sending...
I20260924 17:43:40.439433 7157 ts_manager.cc:194] Re-registered known tserver with Master: 05048dd9ae124416b36034608ad7f8d7 (127.1.8.131:43985)
I20260924 17:43:40.439604 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.426424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.1.8.189:45075)
0924 17:43:40.428052 (+ 1628us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.428054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.428061 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:43:40.428081 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.428081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.428092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.428148 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.428812 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.429554 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.432854 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.433182 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.433247 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.434824 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.434825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.434826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.434832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.436554 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.436933 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.436934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.436934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.436993 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.438665 (+ 1672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.438666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.438666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.438811 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.438928 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.439386 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.439477 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1600,"thread_start_us":62,"threads_started":1}
I20260924 17:43:40.440012 7157 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:59305
I20260924 17:43:40.440197 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.425847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.1.8.190:39843)
0924 17:43:40.426317 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.426319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.426454 (+ 135us) server_negotiation.cc:407] Connection header received
0924 17:43:40.426561 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.426562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.426577 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.426628 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.427863 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.428699 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.429740 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.430029 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.430101 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.431637 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.431639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.431639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.431647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.438899 (+ 7252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.439288 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.439289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.439289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.439345 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.439621 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.439622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.439622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.439753 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.439886 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.440001 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.440065 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":81,"threads_started":1}
I20260924 17:43:40.441718 7157 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:43:40.443631 7157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:43:40.451220 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.447431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.1.8.131:43985)
0924 17:43:40.447792 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.447797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.447808 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:43:40.447849 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.447851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.447869 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.447980 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:40.448122 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.448881 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.450096 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.450874 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.450954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.451034 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.451103 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.451140 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":93,"threads_started":1}
I20260924 17:43:40.451220 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.447287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:43985 (local address 127.0.0.1:33938)
0924 17:43:40.447351 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.447364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.447415 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.447976 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.447976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.447979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:40.448045 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.448047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.448958 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.448959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.449978 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.449980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.450951 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.450952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.451009 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.451013 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.451087 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.451130 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 17:43:40.451781 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.447422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:45267 (local address 127.0.0.1:55206)
0924 17:43:40.447553 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.447566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.447605 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.448117 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.448118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.448120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:40.448173 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.448175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.449611 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.449612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.450578 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.450581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.451540 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.451541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.451603 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.451607 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.451637 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.451681 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260924 17:43:40.452174 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.447747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.1.8.129:45267)
0924 17:43:40.447950 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.447955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.447964 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:40.447997 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.447999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.448013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.448256 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:40.448315 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.449519 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.450709 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.451447 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.451923 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.452009 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.452037 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.452081 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
I20260924 17:43:40.452279 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.447296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:34319 (local address 127.0.0.1:54504)
0924 17:43:40.447551 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.447560 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.447904 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.448089 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.448090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.448092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:40.448123 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.448124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.449402 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.449403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.450618 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.450621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.451808 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.451809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.452113 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.452118 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.452164 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.452206 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":85}
I20260924 17:43:40.452319 8327 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdd97220f9c43e69c54d25a50eb6dc7 (DEFAULT_TABLE table=test-table [id=180159ea1ed54197acee8a3233f37d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:40.452577 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.447505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.1.8.130:34319)
0924 17:43:40.447933 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.447938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.447948 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:43:40.447990 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.447992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.448009 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.448097 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:40.448181 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.449321 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.450697 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.451604 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.451665 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.451736 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.452407 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.452451 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":107,"threads_started":1}
I20260924 17:43:40.452724 8327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdd97220f9c43e69c54d25a50eb6dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:40.452855 8054 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdd97220f9c43e69c54d25a50eb6dc7 (DEFAULT_TABLE table=test-table [id=180159ea1ed54197acee8a3233f37d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:40.453167 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdd97220f9c43e69c54d25a50eb6dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:40.453228 8191 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdd97220f9c43e69c54d25a50eb6dc7 (DEFAULT_TABLE table=test-table [id=180159ea1ed54197acee8a3233f37d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:43:40.453542 8191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdd97220f9c43e69c54d25a50eb6dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:43:40.456570 8418 tablet_bootstrap.cc:492] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Bootstrap starting.
I20260924 17:43:40.456700 8419 tablet_bootstrap.cc:492] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Bootstrap starting.
I20260924 17:43:40.457259 8418 tablet_bootstrap.cc:654] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:40.457255 8420 tablet_bootstrap.cc:492] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Bootstrap starting.
I20260924 17:43:40.457315 8419 tablet_bootstrap.cc:654] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:40.457579 8418 log.cc:824] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:40.457623 8419 log.cc:824] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:40.457885 8420 tablet_bootstrap.cc:654] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Neither blocks nor log segments found. Creating new log.
I20260924 17:43:40.458197 8420 log.cc:824] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Log is configured to *not* fsync() on all Append() calls
I20260924 17:43:40.458367 8419 tablet_bootstrap.cc:492] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: No bootstrap required, opened a new log
I20260924 17:43:40.458501 8419 ts_tablet_manager.cc:1397] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:43:40.459005 8418 tablet_bootstrap.cc:492] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: No bootstrap required, opened a new log
I20260924 17:43:40.459005 8420 tablet_bootstrap.cc:492] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: No bootstrap required, opened a new log
I20260924 17:43:40.459079 8420 ts_tablet_manager.cc:1397] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 17:43:40.459112 8418 ts_tablet_manager.cc:1397] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:43:40.460065 8419 raft_consensus.cc:348] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.460227 8419 raft_consensus.cc:374] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:40.460266 8419 raft_consensus.cc:729] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05048dd9ae124416b36034608ad7f8d7, State: Initialized, Role: FOLLOWER
I20260924 17:43:40.460443 8419 consensus_queue.cc:260] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.460698 8419 ts_tablet_manager.cc:1428] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:43:40.461061 8394 heartbeater.cc:503] Master 127.1.8.190:39843 was elected leader, sending a full tablet report...
I20260924 17:43:40.461604 8420 raft_consensus.cc:348] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.462190 8420 raft_consensus.cc:374] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:40.462224 8420 raft_consensus.cc:729] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea480d5a2def493c81aac3c1599ef1d1, State: Initialized, Role: FOLLOWER
I20260924 17:43:40.462136 8418 raft_consensus.cc:348] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.462267 8418 raft_consensus.cc:374] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:43:40.462306 8418 raft_consensus.cc:729] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60e07ad0b9944a986191eab56f132af, State: Initialized, Role: FOLLOWER
I20260924 17:43:40.462349 8420 consensus_queue.cc:260] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.462432 8418 consensus_queue.cc:260] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.462677 8121 heartbeater.cc:503] Master 127.1.8.190:39843 was elected leader, sending a full tablet report...
I20260924 17:43:40.463055 8420 ts_tablet_manager.cc:1428] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260924 17:43:40.463088 8418 ts_tablet_manager.cc:1428] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 17:43:40.463207 8258 heartbeater.cc:503] Master 127.1.8.190:39843 was elected leader, sending a full tablet report...
I20260924 17:43:40.463227 8424 raft_consensus.cc:482] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:43:40.463289 8424 raft_consensus.cc:504] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.463599 8424 leader_election.cc:288] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60e07ad0b9944a986191eab56f132af (127.1.8.130:34319), ea480d5a2def493c81aac3c1599ef1d1 (127.1.8.129:45267)
I20260924 17:43:40.467671 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.463872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:34319 (local address 127.1.8.131:39027)
0924 17:43:40.463954 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.463966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.464015 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.464504 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.464505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.464507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:40.464878 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.464881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.465660 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.465661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.466819 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.466822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.467483 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.467484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.467545 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.467548 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.467588 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.467618 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 17:43:40.467708 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.464361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39027 (local address 127.1.8.130:34319)
0924 17:43:40.464406 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.464408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.464416 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:43:40.464437 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.464437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.464448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.464499 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:40.464965 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.465577 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.466894 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.467405 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.467481 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.467490 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.467608 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.467644 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:43:40.467998 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd97220f9c43e69c54d25a50eb6dc7" candidate_uuid: "05048dd9ae124416b36034608ad7f8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60e07ad0b9944a986191eab56f132af" is_pre_election: true
I20260924 17:43:40.468070 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.463991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:45267 (local address 127.1.8.131:43741)
0924 17:43:40.464025 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:40.464034 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:43:40.464068 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:43:40.465169 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:43:40.465169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:43:40.465171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:43:40.465243 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.465245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.466058 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.466059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.467048 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:43:40.467050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.467800 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.467802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:43:40.467888 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.467893 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:43:40.467952 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:43:40.467996 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 17:43:40.468070 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.464861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:43741 (local address 127.1.8.129:45267)
0924 17:43:40.464953 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.464955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.464962 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:43:40.464981 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.464982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.465004 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.465072 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:43:40.465310 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.465967 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.467117 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.467690 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.467754 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.467765 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.467968 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.468009 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 17:43:40.468147 8211 raft_consensus.cc:2454] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05048dd9ae124416b36034608ad7f8d7 in term 0.
I20260924 17:43:40.468327 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd97220f9c43e69c54d25a50eb6dc7" candidate_uuid: "05048dd9ae124416b36034608ad7f8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea480d5a2def493c81aac3c1599ef1d1" is_pre_election: true
I20260924 17:43:40.468452 8074 raft_consensus.cc:2454] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05048dd9ae124416b36034608ad7f8d7 in term 0.
I20260924 17:43:40.468677 8282 leader_election.cc:302] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05048dd9ae124416b36034608ad7f8d7, ea480d5a2def493c81aac3c1599ef1d1; no voters:
I20260924 17:43:40.468822 8425 raft_consensus.cc:2781] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:43:40.468904 8425 raft_consensus.cc:482] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:43:40.468948 8425 raft_consensus.cc:3037] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:40.469641 8425 raft_consensus.cc:504] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.469776 8425 leader_election.cc:288] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [CANDIDATE]: Term 1 election: Requested vote from peers c60e07ad0b9944a986191eab56f132af (127.1.8.130:34319), ea480d5a2def493c81aac3c1599ef1d1 (127.1.8.129:45267)
I20260924 17:43:40.469969 8211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd97220f9c43e69c54d25a50eb6dc7" candidate_uuid: "05048dd9ae124416b36034608ad7f8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60e07ad0b9944a986191eab56f132af"
I20260924 17:43:40.469996 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd97220f9c43e69c54d25a50eb6dc7" candidate_uuid: "05048dd9ae124416b36034608ad7f8d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea480d5a2def493c81aac3c1599ef1d1"
I20260924 17:43:40.470057 8074 raft_consensus.cc:3037] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:40.470057 8211 raft_consensus.cc:3037] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:43:40.470999 8211 raft_consensus.cc:2454] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05048dd9ae124416b36034608ad7f8d7 in term 1.
I20260924 17:43:40.470999 8074 raft_consensus.cc:2454] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05048dd9ae124416b36034608ad7f8d7 in term 1.
I20260924 17:43:40.471182 8282 leader_election.cc:302] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05048dd9ae124416b36034608ad7f8d7, ea480d5a2def493c81aac3c1599ef1d1; no voters:
I20260924 17:43:40.471308 8425 raft_consensus.cc:2781] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:43:40.471392 8425 raft_consensus.cc:686] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 1 LEADER]: Becoming Leader. State: Replica: 05048dd9ae124416b36034608ad7f8d7, State: Running, Role: LEADER
I20260924 17:43:40.471472 8425 consensus_queue.cc:237] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } }
I20260924 17:43:40.472535 7157 catalog_manager.cc:5495] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05048dd9ae124416b36034608ad7f8d7 (127.1.8.131). New cstate: current_term: 1 leader_uuid: "05048dd9ae124416b36034608ad7f8d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05048dd9ae124416b36034608ad7f8d7" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 43985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 } health_report { overall_health: UNKNOWN } } }
I20260924 17:43:40.482112 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.479775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.1.8.131:43985)
0924 17:43:40.479835 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.479837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.479846 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:43:40.479891 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.479892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.479911 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.479972 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:43:40.480093 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.480552 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.481348 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.481593 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.481652 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.481792 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:43:40.481895 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:43:40.481956 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.481975 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.481992 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:43:40.484025 8211 raft_consensus.cc:1261] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 1 FOLLOWER]: Refusing update from remote peer 05048dd9ae124416b36034608ad7f8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:40.484025 8074 raft_consensus.cc:1261] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 1 FOLLOWER]: Refusing update from remote peer 05048dd9ae124416b36034608ad7f8d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:43:40.484387 8424 consensus_queue.cc:1035] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60e07ad0b9944a986191eab56f132af" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:40.484601 8425 consensus_queue.cc:1035] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea480d5a2def493c81aac3c1599ef1d1" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:43:40.487644 8430 mvcc.cc:205] Tried to move back new op lower bound from 7332953376699641856 to 7332953376652062720. Current Snapshot: MvccSnapshot[applied={T|T < 7332953376699641856}]
I20260924 17:43:40.488857 8431 mvcc.cc:205] Tried to move back new op lower bound from 7332953376699641856 to 7332953376652062720. Current Snapshot: MvccSnapshot[applied={T|T < 7332953376699641856 or (T in {7332953376699641856})}]
I20260924 17:43:40.488777 8432 mvcc.cc:205] Tried to move back new op lower bound from 7332953376699641856 to 7332953376652062720. Current Snapshot: MvccSnapshot[applied={T|T < 7332953376699641856 or (T in {7332953376699641856})}]
I20260924 17:43:40.491242 7156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:40.491344 7156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:43:40.492767 7156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d79e6ab0487943a99573b2a4e2c379d7: Sending DeleteTablet for 3 replicas of tablet 5cdd97220f9c43e69c54d25a50eb6dc7
I20260924 17:43:40.493125 8327 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdd97220f9c43e69c54d25a50eb6dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:40 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33938
I20260924 17:43:40.493165 8191 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdd97220f9c43e69c54d25a50eb6dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:40 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:54504
I20260924 17:43:40.493371 8054 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdd97220f9c43e69c54d25a50eb6dc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:43:40 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:55206
I20260924 17:43:40.493587 8442 tablet_replica.cc:330] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: stopping tablet replica
I20260924 17:43:40.493695 8442 raft_consensus.cc:2229] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:40.493805 8442 raft_consensus.cc:2258] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:40.493911 8441 tablet_replica.cc:330] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: stopping tablet replica
I20260924 17:43:40.494062 8441 raft_consensus.cc:2229] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:43:40.494194 8441 raft_consensus.cc:2258] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:40.494508 8440 tablet_replica.cc:330] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: stopping tablet replica
I20260924 17:43:40.494598 8442 ts_tablet_manager.cc:1905] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:40.494623 8440 raft_consensus.cc:2229] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:43:40.494762 8441 ts_tablet_manager.cc:1905] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:43:40.494778 8440 raft_consensus.cc:2258] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:43:40.495304 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7125
I20260924 17:43:40.496734 8442 ts_tablet_manager.cc:1918] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:40.496821 8442 log.cc:1196] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/5cdd97220f9c43e69c54d25a50eb6dc7
I20260924 17:43:40.496930 8441 ts_tablet_manager.cc:1918] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:40.496995 8441 log.cc:1196] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/5cdd97220f9c43e69c54d25a50eb6dc7
I20260924 17:43:40.497097 8442 ts_tablet_manager.cc:1939] T 5cdd97220f9c43e69c54d25a50eb6dc7 P ea480d5a2def493c81aac3c1599ef1d1: Deleting consensus metadata
I20260924 17:43:40.497262 8441 ts_tablet_manager.cc:1939] T 5cdd97220f9c43e69c54d25a50eb6dc7 P c60e07ad0b9944a986191eab56f132af: Deleting consensus metadata
I20260924 17:43:40.499625 8440 ts_tablet_manager.cc:1905] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:43:40.501518 8007 connection.cc:535] server connection from 127.0.0.1:55206 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:43:40.501663 8440 ts_tablet_manager.cc:1918] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:43:40.501732 8440 log.cc:1196] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/5cdd97220f9c43e69c54d25a50eb6dc7
I20260924 17:43:40.501998 8440 ts_tablet_manager.cc:1939] T 5cdd97220f9c43e69c54d25a50eb6dc7 P 05048dd9ae124416b36034608ad7f8d7: Deleting consensus metadata
W20260924 17:43:40.502355 8146 connection.cc:535] server connection from 127.0.0.1:54504 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 17:43:40.502697 8280 connection.cc:441] server connection from 127.0.0.1:33938 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33938 (request call id 1) could send its response
I20260924 17:43:40.504029 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.494247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.1.8.188:34117)
0924 17:43:40.494297 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.494299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.495389 (+ 1090us) server_negotiation.cc:407] Connection header received
0924 17:43:40.495500 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.495501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.495516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.495556 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.496635 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.497407 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.498601 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.498983 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.499063 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.500779 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.500780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.500781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.500789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.502597 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.503088 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.503089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.503090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.503155 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.503382 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.503383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.503384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.503529 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.503587 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.503832 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.503927 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:43:40.512765 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.504803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.1.8.189:45075)
0924 17:43:40.504855 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.504857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.504875 (+ 18us) server_negotiation.cc:407] Connection header received
0924 17:43:40.504999 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.505000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.505014 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.505047 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.506154 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.506959 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.507778 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.508097 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.508160 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.509738 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.509740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.509740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.509747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.511367 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.511800 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.511801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.511802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.511884 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.512220 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.512221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.512222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.512358 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.512470 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.512573 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.512657 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 17:43:40.512776 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:40.504933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.1.8.188:34117)
0924 17:43:40.504973 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:43:40.504974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:43:40.505098 (+ 124us) server_negotiation.cc:407] Connection header received
0924 17:43:40.505188 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:43:40.505189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:43:40.505197 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:43:40.505225 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:43:40.505917 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.506573 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.507465 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:43:40.507810 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:43:40.507885 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:43:40.509609 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:43:40.509610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:43:40.509611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:43:40.509623 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:43:40.511290 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.511814 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.511815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.511816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.511888 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:43:40.512118 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:43:40.512120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:43:40.512120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:43:40.512262 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:43:40.512353 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 17:43:40.512554 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 17:43:40.512636 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:43:41.488659 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:41.488341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:56145)
0924 17:43:41.488483 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:41.488580 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
W20260924 17:43:41.488899 8394 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:39843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:41.489100 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:41.488821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:46415)
0924 17:43:41.488946 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:41.489025 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":45,"threads_started":1}
W20260924 17:43:41.489449 8121 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:39843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:41.489475 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:41.489135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:36813)
0924 17:43:41.489276 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:41.489340 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
W20260924 17:43:41.489647 8258 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:39843 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:41.993319 8453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d79e6ab0487943a99573b2a4e2c379d7)
I20260924 17:43:41.993551 8453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:41.993779 8453 leader_election.cc:288] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843)
I20260924 17:43:41.994211 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:41.993996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:57236)
0924 17:43:41.994105 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:41.994170 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
W20260924 17:43:41.994360 7739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:41.994928 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:41.994788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:57240)
0924 17:43:41.994860 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:41.994898 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 17:43:41.995059 7739 leader_election.cc:334] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843): Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:41.995186 7739 leader_election.cc:302] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2db65fc538c949deb183b4b171888aa3; no voters: d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:43:41.995303 8453 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:43:42.489533 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:42.489212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:59291)
0924 17:43:42.489361 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:42.489413 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260924 17:43:42.490106 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:42.489761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:34349)
0924 17:43:42.489984 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:42.490062 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":70,"threads_started":1}
I20260924 17:43:42.490257 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:42.489997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:60815)
0924 17:43:42.490179 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:42.490212 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260924 17:43:43.490585 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:43.490277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:41493)
0924 17:43:43.490475 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:43.490525 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":59,"threads_started":1}
I20260924 17:43:43.491047 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:43.490698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:52345)
0924 17:43:43.490953 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:43.490997 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":83,"threads_started":1}
I20260924 17:43:43.491277 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:43.490959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:42407)
0924 17:43:43.491176 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:43.491212 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
I20260924 17:43:44.198659 8462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d79e6ab0487943a99573b2a4e2c379d7)
I20260924 17:43:44.198761 8462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:44.198962 8462 leader_election.cc:288] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843)
I20260924 17:43:44.199420 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:44.199153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:38128)
0924 17:43:44.199264 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:44.199377 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":45,"threads_started":1}
I20260924 17:43:44.200037 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:44.199911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:38130)
0924 17:43:44.199980 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:44.200013 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 17:43:44.200151 7739 leader_election.cc:334] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843): Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:44.200225 7739 leader_election.cc:302] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2db65fc538c949deb183b4b171888aa3; no voters: d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:43:44.200351 8462 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:43:44.491544 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:44.491244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:46521)
0924 17:43:44.491411 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:44.491458 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260924 17:43:44.491539 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:44.491357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:47057)
0924 17:43:44.491445 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:44.491483 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":44,"threads_started":1}
I20260924 17:43:44.491834 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:44.491652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:38591)
0924 17:43:44.491752 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:44.491792 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260924 17:43:45.492384 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:45.492058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:60845)
0924 17:43:45.492283 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:45.492327 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":61,"threads_started":1}
I20260924 17:43:45.492614 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:45.492105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:54141)
0924 17:43:45.492526 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:45.492563 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":66,"threads_started":1}
I20260924 17:43:45.492759 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:45.492317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:54279)
0924 17:43:45.492584 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:45.492618 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":44,"threads_started":1}
I20260924 17:43:46.025941 8471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d79e6ab0487943a99573b2a4e2c379d7)
I20260924 17:43:46.026047 8471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:46.026257 8471 leader_election.cc:288] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843)
I20260924 17:43:46.026692 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:46.026495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:38340)
0924 17:43:46.026621 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:46.026656 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260924 17:43:46.027125 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:46.027025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:38346)
0924 17:43:46.027077 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:46.027105 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 17:43:46.027243 7739 leader_election.cc:334] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843): Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:46.027307 7739 leader_election.cc:302] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2db65fc538c949deb183b4b171888aa3; no voters: d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:43:46.027397 8471 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:43:46.493323 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:46.492830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:58741)
0924 17:43:46.493213 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:46.493262 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":87,"threads_started":1}
I20260924 17:43:46.493445 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:46.493212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:56381)
0924 17:43:46.493380 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:46.493407 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":59,"threads_started":1}
I20260924 17:43:46.493445 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:46.493211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:45933)
0924 17:43:46.493359 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:46.493400 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":54,"threads_started":1}
I20260924 17:43:47.494333 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:47.494072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:46601)
0924 17:43:47.494233 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:47.494274 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260924 17:43:47.494858 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:47.494059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:43889)
0924 17:43:47.494738 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:47.494809 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":68,"threads_started":1}
I20260924 17:43:47.494865 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:47.494232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:36607)
0924 17:43:47.494773 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:47.494824 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":226,"threads_started":1}
I20260924 17:43:48.495253 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:48.494794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:38607)
0924 17:43:48.495139 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:48.495188 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":76,"threads_started":1}
I20260924 17:43:48.495612 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:48.495316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:51111)
0924 17:43:48.495513 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:48.495555 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260924 17:43:48.495857 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:48.495536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:35941)
0924 17:43:48.495771 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:48.495809 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":67,"threads_started":1}
I20260924 17:43:49.496227 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:49.495938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:39273)
0924 17:43:49.496114 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:49.496162 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260924 17:43:49.496323 8484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:49.496007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:47907)
0924 17:43:49.496211 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:49.496248 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":42,"threads_started":1}
I20260924 17:43:49.496569 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:49.496389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:38947)
0924 17:43:49.496495 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:49.496524 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":46,"threads_started":1}
I20260924 17:43:50.298696 8486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d79e6ab0487943a99573b2a4e2c379d7)
I20260924 17:43:50.298803 8486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:50.299024 8486 leader_election.cc:288] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843)
I20260924 17:43:50.299476 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:50.299227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:38672)
0924 17:43:50.299393 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:50.299435 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
W20260924 17:43:50.299580 7739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260924 17:43:50.300015 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:50.299876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:38684)
0924 17:43:50.299959 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:50.299992 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 17:43:50.300102 7739 leader_election.cc:334] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843): Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:50.300163 7739 leader_election.cc:302] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2db65fc538c949deb183b4b171888aa3; no voters: d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:43:50.300274 8486 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:43:50.497180 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:50.496875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:35077)
0924 17:43:50.497066 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:50.497116 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260924 17:43:50.497288 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:50.497084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:36347)
0924 17:43:50.497203 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:50.497238 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260924 17:43:50.497380 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:50.497183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:55383)
0924 17:43:50.497291 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:50.497337 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260924 17:43:51.498288 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:51.497779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:57163)
0924 17:43:51.498073 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:51.498117 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":64,"threads_started":1}
I20260924 17:43:51.498288 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:51.498068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:60845)
0924 17:43:51.498211 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:51.498247 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":53,"threads_started":1}
I20260924 17:43:51.498539 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:51.498272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:41647)
0924 17:43:51.498455 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:51.498495 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":45,"threads_started":1}
I20260924 17:43:52.499120 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:52.498795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:34519)
0924 17:43:52.499008 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:52.499060 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260924 17:43:52.499522 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:52.498866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:38595)
0924 17:43:52.499426 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:52.499470 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":56,"threads_started":1}
I20260924 17:43:52.499616 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:52.499351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:43417)
0924 17:43:52.499541 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:52.499573 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260924 17:43:53.500608 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:53.500146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:42425)
0924 17:43:53.500484 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:53.500537 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":150,"threads_started":1}
I20260924 17:43:53.500732 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:53.500101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:35923)
0924 17:43:53.500613 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:53.500675 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":191,"threads_started":1}
I20260924 17:43:53.500826 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:53.500495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:48397)
0924 17:43:53.500700 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:53.500733 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260924 17:43:54.501691 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:54.501428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:49915)
0924 17:43:54.501581 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:54.501630 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260924 17:43:54.501719 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:54.501430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:59325)
0924 17:43:54.501648 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:54.501678 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260924 17:43:54.501860 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:54.501443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:52333)
0924 17:43:54.501719 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:54.501764 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":76,"threads_started":1}
I20260924 17:43:55.502661 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:55.502365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:54067)
0924 17:43:55.502558 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:55.502606 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260924 17:43:55.502708 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:55.502365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:52371)
0924 17:43:55.502641 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:55.502668 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":85,"threads_started":1}
I20260924 17:43:55.503055 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:55.502823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:35647)
0924 17:43:55.502943 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:55.502981 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 17:43:56.503434 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:56.503073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:42179)
0924 17:43:56.503325 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:56.503372 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260924 17:43:56.503684 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:56.503413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:33127)
0924 17:43:56.503594 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:56.503629 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":52,"threads_started":1}
I20260924 17:43:56.503984 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:56.503699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:35759)
0924 17:43:56.503836 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:56.503912 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":50,"threads_started":1}
I20260924 17:43:57.504442 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:57.504201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:47871)
0924 17:43:57.504332 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:57.504380 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":47,"threads_started":1}
I20260924 17:43:57.504456 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:57.504078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:54583)
0924 17:43:57.504359 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:57.504396 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":70,"threads_started":1}
I20260924 17:43:57.504778 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:57.504553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:37565)
0924 17:43:57.504695 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:57.504733 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260924 17:43:57.814538 8513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d79e6ab0487943a99573b2a4e2c379d7)
I20260924 17:43:57.814656 8513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:43:57.814827 8513 leader_election.cc:288] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843)
I20260924 17:43:57.815212 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:57.815010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:40090)
0924 17:43:57.815126 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:57.815171 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
W20260924 17:43:57.815322 7739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:57.815689 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:57.815597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:40106)
0924 17:43:57.815647 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:57.815673 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:43:57.815788 7739 leader_election.cc:334] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843): Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:43:57.815848 7739 leader_election.cc:302] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2db65fc538c949deb183b4b171888aa3; no voters: d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:43:57.815945 8513 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:43:58.505348 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:58.504894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:42253)
0924 17:43:58.505189 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:58.505241 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":64,"threads_started":1}
I20260924 17:43:58.505550 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:58.504888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:35167)
0924 17:43:58.505458 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:58.505498 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":77,"threads_started":1}
I20260924 17:43:58.505649 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:58.505252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:57823)
0924 17:43:58.505494 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:58.505596 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260924 17:43:59.506436 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:59.506003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:52163)
0924 17:43:59.506309 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:59.506353 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":84,"threads_started":1}
I20260924 17:43:59.506624 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:59.506287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:48273)
0924 17:43:59.506529 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:59.506569 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":56,"threads_started":1}
I20260924 17:43:59.506681 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:43:59.506357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:47125)
0924 17:43:59.506578 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 17:43:59.506634 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260924 17:44:00.507246 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:00.507027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:33763)
0924 17:44:00.507139 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:00.507181 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260924 17:44:00.507303 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:00.507083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:52733)
0924 17:44:00.507213 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:00.507251 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
I20260924 17:44:00.507553 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:00.507306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:40787)
0924 17:44:00.507476 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:00.507507 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":48,"threads_started":1}
I20260924 17:44:01.508293 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:01.507791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:52721)
0924 17:44:01.508097 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:01.508142 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":80,"threads_started":1}
I20260924 17:44:01.508311 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:01.507991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:59035)
0924 17:44:01.508208 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:01.508247 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":56,"threads_started":1}
I20260924 17:44:01.508549 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:01.508276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:60791)
0924 17:44:01.508465 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:01.508500 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":44,"threads_started":1}
I20260924 17:44:02.509186 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:02.508733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:46081)
0924 17:44:02.509074 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:02.509124 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":64,"threads_started":1}
I20260924 17:44:02.509344 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:02.508827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:46547)
0924 17:44:02.509192 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:02.509297 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":58,"threads_started":1}
I20260924 17:44:02.509574 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:02.509126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:47953)
0924 17:44:02.509498 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:02.509534 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":50,"threads_started":1}
I20260924 17:44:03.510030 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:03.509726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:57439)
0924 17:44:03.509936 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:03.509976 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260924 17:44:03.510318 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:03.509869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:42805)
0924 17:44:03.510229 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:03.510270 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":56,"threads_started":1}
I20260924 17:44:03.510648 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:03.510467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:48663)
0924 17:44:03.510569 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:03.510602 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
I20260924 17:44:04.511052 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:04.510623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:33543)
0924 17:44:04.510908 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:04.510991 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":93,"threads_started":1}
I20260924 17:44:04.511170 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:04.510872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:44503)
0924 17:44:04.511087 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:04.511124 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":49,"threads_started":1}
I20260924 17:44:04.511530 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:04.511287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:38913)
0924 17:44:04.511446 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:04.511483 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":49,"threads_started":1}
I20260924 17:44:04.610677 8537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d79e6ab0487943a99573b2a4e2c379d7)
I20260924 17:44:04.610775 8537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d79e6ab0487943a99573b2a4e2c379d7" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39843 } } peers { permanent_uuid: "2db65fc538c949deb183b4b171888aa3" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45075 } attrs { promote: false } } peers { permanent_uuid: "5881523ba56c4e8aa460ae226e536ab9" member_type: NON_VOTER last_known_addr { host: "127.1.8.188" port: 34117 } attrs { promote: true } }
I20260924 17:44:04.610929 8537 leader_election.cc:288] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843)
I20260924 17:44:04.611325 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:04.611079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:33808)
0924 17:44:04.611191 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:04.611279 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
W20260924 17:44:04.611441 7739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:44:04.611845 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:04.611749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.0.0.1:33816)
0924 17:44:04.611798 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:04.611825 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:44:04.611925 7739 leader_election.cc:334] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d79e6ab0487943a99573b2a4e2c379d7 (127.1.8.190:39843): Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
I20260924 17:44:04.611989 7739 leader_election.cc:302] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2db65fc538c949deb183b4b171888aa3; no voters: d79e6ab0487943a99573b2a4e2c379d7
I20260924 17:44:04.612080 8537 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2db65fc538c949deb183b4b171888aa3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:44:05.512059 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:05.511825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:39083)
0924 17:44:05.511951 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:05.511994 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260924 17:44:05.512089 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:05.511827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:45697)
0924 17:44:05.511995 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:05.512032 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260924 17:44:05.512427 8542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:05.512079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:54007)
0924 17:44:05.512355 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:05.512388 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":139,"threads_started":1}
I20260924 17:44:06.512912 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:06.512635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:37811)
0924 17:44:06.512815 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:06.512853 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260924 17:44:06.512909 8544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:06.512688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:38509)
0924 17:44:06.512813 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:06.512853 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":52,"threads_started":1}
I20260924 17:44:06.513180 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:06.512889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:54343)
0924 17:44:06.513064 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:06.513112 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":44,"threads_started":1}
I20260924 17:44:07.513825 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:07.513371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:44763)
0924 17:44:07.513722 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:07.513766 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":68,"threads_started":1}
I20260924 17:44:07.513826 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:07.513444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:47345)
0924 17:44:07.513726 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:07.513771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":54,"threads_started":1}
I20260924 17:44:07.513929 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:07.513688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:36829)
0924 17:44:07.513848 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:07.513883 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":46,"threads_started":1}
I20260924 17:44:08.514721 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:08.514428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:37963)
0924 17:44:08.514626 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:08.514664 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
I20260924 17:44:08.514725 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:08.514435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:41953)
0924 17:44:08.514635 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:08.514676 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":58,"threads_started":1}
I20260924 17:44:08.514755 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:08.514432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:50111)
0924 17:44:08.514682 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:08.514712 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":60,"threads_started":1}
I20260924 17:44:09.515523 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:09.515227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:42181)
0924 17:44:09.515417 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:09.515462 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
I20260924 17:44:09.515671 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:09.515433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:45945)
0924 17:44:09.515581 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:09.515623 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 17:44:09.515825 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:09.515479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:54651)
0924 17:44:09.515746 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:09.515782 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":67,"threads_started":1}
I20260924 17:44:10.516492 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:10.516152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.130:52879)
0924 17:44:10.516387 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:10.516430 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260924 17:44:10.516538 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:10.516288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.129:56649)
0924 17:44:10.516470 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:10.516496 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":46,"threads_started":1}
I20260924 17:44:10.516582 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:10.516282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39843 (local address 127.1.8.131:32959)
0924 17:44:10.516457 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:10.516494 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:39843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":54,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.8.190:39843,127.1.8.189:45075,127.1.8.188:34117, num_attempts: 1)
I20260924 17:44:10.748078 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7901
I20260924 17:44:10.754109 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 8128
I20260924 17:44:10.759047 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 8264
I20260924 17:44:10.764736 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7621
I20260924 17:44:10.769523 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 7808
2026-09-24T17:44:10Z chronyd exiting
I20260924 17:44:10.785768 1058 test_util.cc:175] -----------------------------------------------
I20260924 17:44:10.785830 1058 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790271791394214-1058-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33277 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:44:10 dist-test-slave-r7hf krb5kdc[8560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:44:10 dist-test-slave-r7hf krb5kdc[8560](info): set up 2 sockets
Sep 24 17:44:10 dist-test-slave-r7hf krb5kdc[8560](info): commencing operation
krb5kdc: starting...
W20260924 17:44:11.803143 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:44:11 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab.
Entry for principal oryx/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab.
I20260924 17:44:11.925341 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41207
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab
--principal=oryx/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:12.009433 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:12.009671 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:12.009696 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:12.010982 8573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:12.011036 8573 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 17:44:12.011050 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:12.011065 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:12.011077 8573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:12.011089 8573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:12.012616 8573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:41207
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.190.keytab
--principal=oryx/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:12.012823 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:12.013005 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:12.017926 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.018105 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.017926 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:12.018728 8573 server_base.cc:1034] running on GCE node
I20260924 17:44:12.019060 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:12.019434 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:12.020586 8573 hybrid_clock.cc:648] HybridClock initialized: now 1790271852020584 us; error 60 us; skew 500 ppm
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:12.023244 8573 init.cc:377] Logged in from keytab as oryx/127.1.8.190@KRBTEST.COM (short username oryx)
I20260924 17:44:12.023787 8573 webserver.cc:460] Webserver started at http://127.1.8.190:37973/ using document root <none> and password file <none>
I20260924 17:44:12.024025 8573 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:12.024097 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:12.024257 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:12.025386 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "e6881a2b7d5b4585aeccd2c54b62a309"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "c55a1ce60ae43962814000ab8039287a"
server_key_iv: "40eea487053734ae61d1e02e0a976e78"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.025765 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "e6881a2b7d5b4585aeccd2c54b62a309"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "c55a1ce60ae43962814000ab8039287a"
server_key_iv: "40eea487053734ae61d1e02e0a976e78"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.027271 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:44:12.028010 8588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.028234 8573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:12.028349 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "e6881a2b7d5b4585aeccd2c54b62a309"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "c55a1ce60ae43962814000ab8039287a"
server_key_iv: "40eea487053734ae61d1e02e0a976e78"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.028429 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:12.044283 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:12.046201 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:12.046376 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:12.051517 8573 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:41207
I20260924 17:44:12.051574 8650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:41207 every 8 connection(s)
I20260924 17:44:12.051947 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:44:12.052646 8651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.055105 8651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Bootstrap starting.
I20260924 17:44:12.055754 8651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.056061 8651 log.cc:824] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:12.056782 8651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: No bootstrap required, opened a new log
I20260924 17:44:12.058943 8651 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } }
I20260924 17:44:12.059101 8651 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.059125 8651 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6881a2b7d5b4585aeccd2c54b62a309, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.059350 8651 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } }
I20260924 17:44:12.059453 8654 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:12.059557 8654 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } }
I20260924 17:44:12.059423 8651 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:44:12.059696 8654 leader_election.cc:302] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6881a2b7d5b4585aeccd2c54b62a309; no voters:
I20260924 17:44:12.059710 8651 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:44:12.059798 8651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.059870 8654 leader_election.cc:288] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260924 17:44:12.060415 8651 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } }
I20260924 17:44:12.060513 8651 leader_election.cc:302] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6881a2b7d5b4585aeccd2c54b62a309; no voters:
I20260924 17:44:12.060565 8651 leader_election.cc:288] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:44:12.060572 8655 raft_consensus.cc:2741] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 17:44:12.060745 8655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:12.060801 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 8573
I20260924 17:44:12.060820 8655 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [term 1 LEADER]: Becoming Leader. State: Replica: e6881a2b7d5b4585aeccd2c54b62a309, State: Running, Role: LEADER
I20260924 17:44:12.060882 8651 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:44:12.060899 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:44:12.060920 8655 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } }
I20260924 17:44:12.061100 1058 external_mini_cluster.cc:1428] Setting key ef7036cc20ce1348ab6a2a81aa130250
I20260924 17:44:12.061333 8655 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6881a2b7d5b4585aeccd2c54b62a309. Latest consensus state: current_term: 1 leader_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } } }
I20260924 17:44:12.061482 8655 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:12.061430 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6881a2b7d5b4585aeccd2c54b62a309" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41207 } } }
I20260924 17:44:12.061728 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:12.061903 8664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:44:12.062753 8664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:44:12.064358 8664 catalog_manager.cc:1347] Generated new cluster ID: e09a9d91e62c406dbc2d36b08dab2c4b
I20260924 17:44:12.064410 8664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:12.072077 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.061807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.1.8.190:41207)
0924 17:44:12.062064 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.062074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.062111 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:44:12.062239 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.062245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.062304 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.063344 (+ 1040us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.063452 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.064631 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.065397 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.065854 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.065983 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.068400 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.068404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.068405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.068425 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.070197 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.070655 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.070658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.070659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.070776 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.070999 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.071000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.071000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.071282 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.071462 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.071533 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.071827 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260924 17:44:12.100005 8664 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:44:12.100781 8664 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:44:12.107200 8664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Generated new TSK 0
I20260924 17:44:12.107419 8664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab.
Entry for principal oryx/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab.
I20260924 17:44:12.141801 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41207
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:12.227099 8677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:12.227316 8677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:12.227344 8677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:12.228562 8677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:12.228610 8677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:12.228632 8677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:44:12.228698 8677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:44:12.230304 8677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:41207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:12.230592 8677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:12.230847 8677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:44:12.236092 8677 server_base.cc:1034] running on GCE node
W20260924 17:44:12.236047 8685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.236017 8683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.236033 8682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:12.236445 8677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:12.236752 8677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:12.237918 8677 hybrid_clock.cc:648] HybridClock initialized: now 1790271852237906 us; error 45 us; skew 500 ppm
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:12.240497 8677 init.cc:377] Logged in from keytab as oryx/127.1.8.129@KRBTEST.COM (short username oryx)
I20260924 17:44:12.240954 8677 webserver.cc:460] Webserver started at http://127.1.8.129:43373/ using document root <none> and password file <none>
I20260924 17:44:12.241219 8677 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:12.241294 8677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:12.241446 8677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:12.242622 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "2898c857d10341fe86d401c94d7ac286"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "43d7563c15ca0c67ae677eabf33029f6"
server_key_iv: "e56501614aaa6cec53d8dd8d22ec2b30"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.242991 8677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "2898c857d10341fe86d401c94d7ac286"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "43d7563c15ca0c67ae677eabf33029f6"
server_key_iv: "e56501614aaa6cec53d8dd8d22ec2b30"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.244349 8677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:12.245111 8692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.245309 8677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:12.245415 8677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "2898c857d10341fe86d401c94d7ac286"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "43d7563c15ca0c67ae677eabf33029f6"
server_key_iv: "e56501614aaa6cec53d8dd8d22ec2b30"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.245494 8677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:12.260944 8677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:12.263432 8677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:12.263614 8677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:12.264174 8677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:12.264241 8677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.264289 8677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:12.264318 8677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:12.271757 8677 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:39661
I20260924 17:44:12.272184 8807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:39661 every 8 connection(s)
I20260924 17:44:12.272336 8677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:44:12.274740 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.264729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:40805 (local address 127.1.8.190:41207)
0924 17:44:12.264802 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.264804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.265046 (+ 242us) server_negotiation.cc:407] Connection header received
0924 17:44:12.265610 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.265611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.265630 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.265712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.266695 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.267205 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.268252 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.268463 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.268545 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.271840 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.271842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.271842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.271856 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.273439 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.273826 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.273827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.273827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.273880 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.274090 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.274091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.274091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.274205 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.274265 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.274580 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.274650 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:44:12.274894 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.264704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.1.8.129:40805)
0924 17:44:12.264910 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.264951 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.265477 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.265804 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.265808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.266231 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.266583 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.266589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.267302 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.267303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.268141 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.268143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.268551 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.268553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.268705 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.269300 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.269312 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.271708 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.273516 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.273520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.273522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.273754 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.273939 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.273939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.273940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.274034 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.274272 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.274276 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.274457 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.274563 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.274695 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":132,"threads_started":1}
I20260924 17:44:12.275421 8808 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41207
I20260924 17:44:12.275507 8808 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:12.275718 8808 heartbeater.cc:511] Master 127.1.8.190:41207 requested a full tablet report, sending...
I20260924 17:44:12.276252 8605 ts_manager.cc:194] Registered new tserver with Master: 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661)
I20260924 17:44:12.276800 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 8677
I20260924 17:44:12.276906 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:44:12.277117 1058 external_mini_cluster.cc:1428] Setting key 69fd7c163fe0264d844d5481d91a03dc
I20260924 17:44:12.277196 8605 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:40805
I20260924 17:44:12.283428 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.276529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.1.8.129:49247)
0924 17:44:12.276653 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.276693 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.276778 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.276939 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.276940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.277136 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.277230 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.277232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.278124 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.278124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.278710 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.278712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.279222 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.279223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.279297 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.279740 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.279745 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.280196 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.282099 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.282101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.282101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.282350 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.282623 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.282624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.282625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.282709 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.283022 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.283023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.283082 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.283164 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.283292 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260924 17:44:12.283432 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.276607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49247 (local address 127.1.8.190:41207)
0924 17:44:12.276651 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.276653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.276762 (+ 109us) server_negotiation.cc:407] Connection header received
0924 17:44:12.276836 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.276837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.276847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.276897 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.277333 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.278045 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.278788 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.279127 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.279204 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.280297 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.280298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.280299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.280305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.281998 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.282459 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.282460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.282461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.282527 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.282788 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.282789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.282789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.282942 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.283018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.283206 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.283302 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab.
Entry for principal oryx/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab.
I20260924 17:44:12.306130 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41207
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:12.389351 8817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:12.389606 8817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:12.389647 8817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:12.390892 8817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:12.390959 8817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:12.390995 8817 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:44:12.391036 8817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:44:12.392584 8817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:41207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:12.392876 8817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:12.393121 8817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:12.398643 8822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.398643 8823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.398741 8825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:12.398934 8817 server_base.cc:1034] running on GCE node
I20260924 17:44:12.399173 8817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:12.399435 8817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:12.400602 8817 hybrid_clock.cc:648] HybridClock initialized: now 1790271852400596 us; error 36 us; skew 500 ppm
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:12.403285 8817 init.cc:377] Logged in from keytab as oryx/127.1.8.130@KRBTEST.COM (short username oryx)
I20260924 17:44:12.403752 8817 webserver.cc:460] Webserver started at http://127.1.8.130:41981/ using document root <none> and password file <none>
I20260924 17:44:12.403990 8817 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:12.404066 8817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:12.404217 8817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:12.405351 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "44ddc6ab2946434daffc23c828accc39"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "9ceb7af890345bdefb1bbb28e502ec64"
server_key_iv: "03bcb2ede7ab07bd01278fe66f7ca196"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.405737 8817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "44ddc6ab2946434daffc23c828accc39"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "9ceb7af890345bdefb1bbb28e502ec64"
server_key_iv: "03bcb2ede7ab07bd01278fe66f7ca196"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.407284 8817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:44:12.408021 8832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.408205 8817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:12.408309 8817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "44ddc6ab2946434daffc23c828accc39"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "9ceb7af890345bdefb1bbb28e502ec64"
server_key_iv: "03bcb2ede7ab07bd01278fe66f7ca196"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.408386 8817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:12.434032 8817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:12.436062 8817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:12.436288 8817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:12.436854 8817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:12.436918 8817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.436964 8817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:12.436987 8817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:12.444408 8817 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:36699
I20260924 17:44:12.444464 8947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:36699 every 8 connection(s)
I20260924 17:44:12.444785 8817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:44:12.447523 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.437611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:48491 (local address 127.1.8.190:41207)
0924 17:44:12.437676 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.437678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.437896 (+ 218us) server_negotiation.cc:407] Connection header received
0924 17:44:12.438658 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.438660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.438673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.438742 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.439677 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.440458 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.441829 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.442050 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.442144 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.444816 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.444817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.444818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.444827 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.446189 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.446712 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.446713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.446714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.446765 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.446979 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.446979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.446979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.447082 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.447129 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.447395 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.447458 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:44:12.447707 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.437595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.1.8.130:48491)
0924 17:44:12.437768 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.437806 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.438503 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.438836 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.438841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.439255 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.439582 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.439589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.440590 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.440591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.441708 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.441711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.442169 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.442170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.442309 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.443045 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.443055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.444691 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.446285 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.446292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.446293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.446640 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.446811 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.446812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.446812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.446918 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.447141 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.447146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.447275 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.447359 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.447488 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260924 17:44:12.448324 8948 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41207
I20260924 17:44:12.448448 8948 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:12.448642 8948 heartbeater.cc:511] Master 127.1.8.190:41207 requested a full tablet report, sending...
I20260924 17:44:12.449052 8604 ts_manager.cc:194] Registered new tserver with Master: 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699)
I20260924 17:44:12.449559 8604 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:48491
I20260924 17:44:12.451426 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 8817
I20260924 17:44:12.451524 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:44:12.451776 1058 external_mini_cluster.cc:1428] Setting key b6c150d2ba1e71f4d1319102cf28c64e
I20260924 17:44:12.456022 8954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.449590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.1.8.130:47295)
0924 17:44:12.449802 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.449818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.449888 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.450006 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.450007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.450198 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.450247 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.450249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.450933 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.450934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.451446 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.451447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.451920 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.451922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.452005 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.452566 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.452572 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.453021 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.454967 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.454968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.454969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.455153 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.455362 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.455362 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.455363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.455417 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.455754 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.455755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.455800 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.455857 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.455939 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260924 17:44:12.456120 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.449682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:47295 (local address 127.1.8.190:41207)
0924 17:44:12.449724 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.449726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.449873 (+ 147us) server_negotiation.cc:407] Connection header received
0924 17:44:12.449944 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.449945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.449954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.450003 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.450325 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.450837 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.451562 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.451791 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.451872 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.453205 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.453206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.453207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.453213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.454884 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.455227 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.455228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.455229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.455292 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.455478 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.455479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.455479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.455686 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.455758 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.455899 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.455982 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for oryx/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab.
Entry for principal oryx/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab.
I20260924 17:44:12.482116 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41207
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:37163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:12.564688 8957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:12.564906 8957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:12.564931 8957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:12.566121 8957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:12.566171 8957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:12.566192 8957 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 17:44:12.566210 8957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:44:12.567807 8957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:37163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.8.190:41207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:12.568071 8957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:12.568311 8957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:12.573424 8965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.573422 8963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:12.573498 8962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:12.573747 8957 server_base.cc:1034] running on GCE node
I20260924 17:44:12.573961 8957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:12.574215 8957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:12.575443 8957 hybrid_clock.cc:648] HybridClock initialized: now 1790271852575410 us; error 42 us; skew 500 ppm
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:12.578223 8957 init.cc:377] Logged in from keytab as oryx/127.1.8.131@KRBTEST.COM (short username oryx)
I20260924 17:44:12.578714 8957 webserver.cc:460] Webserver started at http://127.1.8.131:40955/ using document root <none> and password file <none>
I20260924 17:44:12.578900 8957 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:12.578939 8957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:12.579028 8957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:12.580067 8957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "b98357dba68b41b496e82d5d591b8b45"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "8ad677b7747d5d5ad09ca73455719285"
server_key_iv: "b79d43e5b0e9000206086d8a6aa4c29c"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.580374 8957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "b98357dba68b41b496e82d5d591b8b45"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "8ad677b7747d5d5ad09ca73455719285"
server_key_iv: "b79d43e5b0e9000206086d8a6aa4c29c"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.581692 8957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:44:12.582492 8972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.582676 8957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:12.582785 8957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "b98357dba68b41b496e82d5d591b8b45"
format_stamp: "Formatted at 2026-09-24 17:44:12 on dist-test-slave-r7hf"
server_key: "8ad677b7747d5d5ad09ca73455719285"
server_key_iv: "b79d43e5b0e9000206086d8a6aa4c29c"
server_key_version: "encryptionkey@0"
I20260924 17:44:12.582860 8957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:12.615444 8957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:12.617558 8957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:12.617748 8957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:12.618311 8957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:12.618379 8957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.618438 8957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:12.618487 8957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.625186 8957 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:45851
I20260924 17:44:12.625303 9087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:45851 every 8 connection(s)
I20260924 17:44:12.625648 8957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:12.627547 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 8957
I20260924 17:44:12.627660 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:44:12.627846 1058 external_mini_cluster.cc:1428] Setting key a0fc5d9d5e577770fab68d1e7f5bb8af
I20260924 17:44:12.629631 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.619097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:46961 (local address 127.1.8.190:41207)
0924 17:44:12.619148 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.619150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.619411 (+ 261us) server_negotiation.cc:407] Connection header received
0924 17:44:12.620108 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.620110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.620123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.620205 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.621123 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.621965 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.623544 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.623888 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.623976 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.626783 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.626785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.626785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.626793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.628224 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.628643 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.628645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.628646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.628708 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.628950 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.628951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.628951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.629074 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.629175 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.629446 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.629528 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:44:12.629854 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.619108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.1.8.131:46961)
0924 17:44:12.619280 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.619318 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.619964 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.620301 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.620307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.620707 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.621011 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.621017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.622082 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.622083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.623443 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.623446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.623990 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.623991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.624141 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.624797 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.624806 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.626682 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.628322 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.628330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.628332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.628550 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.628774 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.628775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.628775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.628880 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.629165 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.629170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.629314 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.629425 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.629612 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260924 17:44:12.630378 9088 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41207
I20260924 17:44:12.630481 9088 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:12.630672 9088 heartbeater.cc:511] Master 127.1.8.190:41207 requested a full tablet report, sending...
I20260924 17:44:12.631075 8605 ts_manager.cc:194] Registered new tserver with Master: b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:12.631448 8605 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:46961
I20260924 17:44:12.636518 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.631544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:42679 (local address 127.1.8.190:41207)
0924 17:44:12.631646 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.631648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.631692 (+ 44us) server_negotiation.cc:407] Connection header received
0924 17:44:12.631746 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.631747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.631758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.631799 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.632166 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.632646 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.633335 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.633504 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.633550 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.634506 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.634506 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.634507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.634511 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.635652 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.635953 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.635954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.635954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.635999 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.636167 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.636167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.636168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.636256 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.636304 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.636405 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.636457 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 17:44:12.636539 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.631498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.1.8.131:42679)
0924 17:44:12.631633 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.631643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.631700 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.631814 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.631815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.632011 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.632069 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.632070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.632741 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.632742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.633264 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.633266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.633578 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.633578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.633627 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.634089 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.634094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.634382 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.635729 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.635730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.635730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.635889 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.636053 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.636054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.636054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.636115 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.636308 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.636309 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.636341 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.636391 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.636465 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260924 17:44:12.639394 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:44:12.661329 8665 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:12.655675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.1.8.190:41207)
0924 17:44:12.655741 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.655743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.655754 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:44:12.655830 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.655831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.655846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.655921 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.656488 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.656979 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.657763 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.657955 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.658044 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.661245 (+ 3201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54380: BlockingRecv error: recv got EOF from 127.0.0.1:54380 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:12.669937 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.662438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.1.8.190:41207)
0924 17:44:12.662497 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.662499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.662613 (+ 114us) server_negotiation.cc:407] Connection header received
0924 17:44:12.662699 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.662700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.662711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.662788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.663295 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.663812 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.664564 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.664763 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.664850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.667280 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.667282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.667283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.667291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.668764 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.669145 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.669146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.669146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.669208 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.669419 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.669420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.669421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.669539 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.669648 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.669744 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.669827 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 17:44:12.671835 8605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:44:12.673590 8605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:44:12.680441 9111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.676303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45851 (local address 127.0.0.1:37284)
0924 17:44:12.676478 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.676489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.676543 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.676816 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.676819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.676821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.676893 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.676895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.678006 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.678007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.679351 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.679354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.680121 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.680122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.680207 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.680223 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.680297 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.680351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260924 17:44:12.680446 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.676417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.1.8.131:45851)
0924 17:44:12.676600 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.676605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.676615 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:44:12.676692 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.676694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.676708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.676794 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.676965 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.677900 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.679443 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.680022 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.680119 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.680205 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.680317 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.680351 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
I20260924 17:44:12.680744 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.676086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36699 (local address 127.0.0.1:33776)
0924 17:44:12.676397 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.676442 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.676513 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.677080 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.677081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.677083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.677132 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.677133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.678471 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.678472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.679538 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.679540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.680466 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.680467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.680560 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.680564 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.680623 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.680674 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":52,"threads_started":1}
I20260924 17:44:12.680922 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.676269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.1.8.129:39661)
0924 17:44:12.677153 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.677156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.677164 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:44:12.677219 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.677220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.677235 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.677317 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.677427 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.678481 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.679501 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.680257 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.680336 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.680411 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.680792 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.680830 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":107,"threads_started":1}
I20260924 17:44:12.681031 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.676207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.1.8.130:36699)
0924 17:44:12.676920 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.676925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.676934 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:44:12.676973 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.676975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.676989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.677061 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.677218 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.678331 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.679617 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.680311 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.680367 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.680433 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.680885 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.680929 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":85,"threads_started":1}
I20260924 17:44:12.681221 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.676092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39661 (local address 127.0.0.1:42350)
0924 17:44:12.676404 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.676442 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.676512 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.677331 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.677332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.677334 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.677369 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.677370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.678556 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.678557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.679399 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.679402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.680581 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.680582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.680643 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.680647 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.680696 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.680735 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":97,"threads_started":1}
I20260924 17:44:12.681231 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 943b3ca9da9644dc9c41f1e544522453 (DEFAULT_TABLE table=test-table [id=596669469816489e8c97090408146b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:12.681661 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943b3ca9da9644dc9c41f1e544522453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.681914 8878 tablet_service.cc:1467] Processing CreateTablet for tablet 943b3ca9da9644dc9c41f1e544522453 (DEFAULT_TABLE table=test-table [id=596669469816489e8c97090408146b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:12.682099 8742 tablet_service.cc:1467] Processing CreateTablet for tablet 943b3ca9da9644dc9c41f1e544522453 (DEFAULT_TABLE table=test-table [id=596669469816489e8c97090408146b28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:12.682230 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943b3ca9da9644dc9c41f1e544522453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.682452 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943b3ca9da9644dc9c41f1e544522453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.685480 9116 tablet_bootstrap.cc:492] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Bootstrap starting.
I20260924 17:44:12.685949 9117 tablet_bootstrap.cc:492] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Bootstrap starting.
I20260924 17:44:12.686267 9116 tablet_bootstrap.cc:654] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.686275 9118 tablet_bootstrap.cc:492] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Bootstrap starting.
I20260924 17:44:12.686697 9117 tablet_bootstrap.cc:654] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.686776 9116 log.cc:824] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:12.686928 9118 tablet_bootstrap.cc:654] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.687062 9117 log.cc:824] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:12.687266 9118 log.cc:824] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:12.687847 9117 tablet_bootstrap.cc:492] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: No bootstrap required, opened a new log
I20260924 17:44:12.687875 9116 tablet_bootstrap.cc:492] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: No bootstrap required, opened a new log
I20260924 17:44:12.687922 9117 ts_tablet_manager.cc:1397] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:44:12.687954 9116 ts_tablet_manager.cc:1397] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 17:44:12.688160 9118 tablet_bootstrap.cc:492] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: No bootstrap required, opened a new log
I20260924 17:44:12.688267 9118 ts_tablet_manager.cc:1397] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:12.689785 9116 raft_consensus.cc:348] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.689944 9116 raft_consensus.cc:374] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.690002 9116 raft_consensus.cc:729] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddc6ab2946434daffc23c828accc39, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.689993 9117 raft_consensus.cc:348] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.690166 9117 raft_consensus.cc:374] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.690209 9117 raft_consensus.cc:729] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98357dba68b41b496e82d5d591b8b45, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.690152 9116 consensus_queue.cc:260] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.690330 9117 consensus_queue.cc:260] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.690256 9118 raft_consensus.cc:348] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.690491 9118 raft_consensus.cc:374] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.690515 9116 ts_tablet_manager.cc:1428] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:12.690553 8948 heartbeater.cc:503] Master 127.1.8.190:41207 was elected leader, sending a full tablet report...
I20260924 17:44:12.690569 9118 raft_consensus.cc:729] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.690685 9118 consensus_queue.cc:260] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.690937 9118 ts_tablet_manager.cc:1428] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:44:12.691066 9117 ts_tablet_manager.cc:1428] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:44:12.691098 9088 heartbeater.cc:503] Master 127.1.8.190:41207 was elected leader, sending a full tablet report...
I20260924 17:44:12.691346 8808 heartbeater.cc:503] Master 127.1.8.190:41207 was elected leader, sending a full tablet report...
W20260924 17:44:12.695659 8949 tablet.cc:2369] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:44:12.703086 9123 raft_consensus.cc:482] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:12.703197 9123 raft_consensus.cc:504] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.703593 9123 leader_election.cc:288] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:12.706794 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.703885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:42489 (local address 127.1.8.131:45851)
0924 17:44:12.703930 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.703932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.704077 (+ 145us) server_negotiation.cc:407] Connection header received
0924 17:44:12.704153 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.704154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.704168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.704230 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.704400 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.704862 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.706019 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.706384 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.706524 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.706533 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.706679 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.706719 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:44:12.706820 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.703879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:38681 (local address 127.1.8.130:36699)
0924 17:44:12.703935 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.703937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.703949 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:44:12.703989 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.703991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.704002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.704079 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.704277 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.704803 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.706023 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.706442 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.706515 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.706526 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.706714 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.706751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:44:12.706848 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.703801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45851 (local address 127.1.8.129:42489)
0924 17:44:12.704004 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.704016 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.704074 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.704256 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.704256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.704258 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.704334 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.704336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.704940 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.704941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.705929 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.705931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.706527 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.706528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.706610 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.706613 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.706655 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.706713 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":23,"thread_start_us":110,"threads_started":1}
I20260924 17:44:12.706864 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.703801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36699 (local address 127.1.8.129:38681)
0924 17:44:12.703868 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.703882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.703926 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.704127 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.704128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.704130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.704205 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.704208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.704886 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.704887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.705926 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.705929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.706521 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.706522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.706614 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.706618 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.706697 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.706737 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":49}
I20260924 17:44:12.707252 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943b3ca9da9644dc9c41f1e544522453" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45" is_pre_election: true
I20260924 17:44:12.707448 9042 raft_consensus.cc:2454] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 0.
I20260924 17:44:12.707417 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943b3ca9da9644dc9c41f1e544522453" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39" is_pre_election: true
I20260924 17:44:12.707592 8902 raft_consensus.cc:2454] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 0.
I20260924 17:44:12.707687 8695 leader_election.cc:302] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, b98357dba68b41b496e82d5d591b8b45; no voters:
I20260924 17:44:12.707840 9123 raft_consensus.cc:2781] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:44:12.707903 9123 raft_consensus.cc:482] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:44:12.707921 9123 raft_consensus.cc:3037] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.708839 9123 raft_consensus.cc:504] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.709012 9123 leader_election.cc:288] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 election: Requested vote from peers 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:12.709136 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943b3ca9da9644dc9c41f1e544522453" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39"
I20260924 17:44:12.709190 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943b3ca9da9644dc9c41f1e544522453" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45"
I20260924 17:44:12.709244 8902 raft_consensus.cc:3037] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.709307 9042 raft_consensus.cc:3037] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.709964 8902 raft_consensus.cc:2454] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 1.
I20260924 17:44:12.710103 9042 raft_consensus.cc:2454] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 1.
I20260924 17:44:12.710218 8696 leader_election.cc:302] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, 44ddc6ab2946434daffc23c828accc39; no voters:
I20260924 17:44:12.710327 9123 raft_consensus.cc:2781] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:12.710594 9123 raft_consensus.cc:686] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 1 LEADER]: Becoming Leader. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Running, Role: LEADER
I20260924 17:44:12.710712 9123 consensus_queue.cc:237] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.711551 8604 catalog_manager.cc:5495] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2898c857d10341fe86d401c94d7ac286 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "2898c857d10341fe86d401c94d7ac286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } health_report { overall_health: UNKNOWN } } }
I20260924 17:44:12.717293 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.714919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42360 (local address 127.1.8.129:39661)
0924 17:44:12.714965 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.714967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.715063 (+ 96us) server_negotiation.cc:407] Connection header received
0924 17:44:12.715088 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.715089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.715105 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.715178 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:12.715289 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.715756 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.716531 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.716722 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.716804 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.716981 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.717080 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.717171 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.717191 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.717211 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:44:12.719121 9042 raft_consensus.cc:1261] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Refusing update from remote peer 2898c857d10341fe86d401c94d7ac286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:12.719174 8902 raft_consensus.cc:1261] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Refusing update from remote peer 2898c857d10341fe86d401c94d7ac286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:12.719455 9123 consensus_queue.cc:1035] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:12.719621 9126 consensus_queue.cc:1035] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:12.721854 9129 mvcc.cc:205] Tried to move back new op lower bound from 7332953508734840832 to 7332953508703662080. Current Snapshot: MvccSnapshot[applied={T|T < 7332953508734840832}]
I20260924 17:44:12.722627 9131 mvcc.cc:205] Tried to move back new op lower bound from 7332953508734840832 to 7332953508703662080. Current Snapshot: MvccSnapshot[applied={T|T < 7332953508734840832}]
I20260924 17:44:12.724030 8604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:12.724136 8604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:12.724862 8604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Sending DeleteTablet for 3 replicas of tablet 943b3ca9da9644dc9c41f1e544522453
I20260924 17:44:12.725250 8742 tablet_service.cc:1514] Processing DeleteTablet for tablet 943b3ca9da9644dc9c41f1e544522453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:12 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260924 17:44:12.725401 8878 tablet_service.cc:1514] Processing DeleteTablet for tablet 943b3ca9da9644dc9c41f1e544522453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:12 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33776
I20260924 17:44:12.725490 9139 tablet_replica.cc:330] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: stopping tablet replica
I20260924 17:44:12.725600 9139 raft_consensus.cc:2229] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:44:12.725651 9140 tablet_replica.cc:330] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: stopping tablet replica
I20260924 17:44:12.725638 9022 tablet_service.cc:1514] Processing DeleteTablet for tablet 943b3ca9da9644dc9c41f1e544522453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:12 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37284
I20260924 17:44:12.725771 9139 raft_consensus.cc:2258] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:12.725795 9140 raft_consensus.cc:2229] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:12.725953 9140 raft_consensus.cc:2258] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:12.726220 8604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 17:44:12.726297 9139 ts_tablet_manager.cc:1905] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:44:12.726370 8604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:44:12.726590 9140 ts_tablet_manager.cc:1905] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:12.727793 8878 tablet_service.cc:1467] Processing CreateTablet for tablet c4afaaedf920406f9dcfd9c7d721e90d (DEFAULT_TABLE table=test-table [id=19b7dbe35c79449e89cba8209d546cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:12.727897 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4afaaedf920406f9dcfd9c7d721e90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.728372 9139 ts_tablet_manager.cc:1918] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:12.728435 9139 log.cc:1196] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/943b3ca9da9644dc9c41f1e544522453
I20260924 17:44:12.728689 9139 ts_tablet_manager.cc:1939] T 943b3ca9da9644dc9c41f1e544522453 P 2898c857d10341fe86d401c94d7ac286: Deleting consensus metadata
I20260924 17:44:12.728981 9140 ts_tablet_manager.cc:1918] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:12.729054 9140 log.cc:1196] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/943b3ca9da9644dc9c41f1e544522453
I20260924 17:44:12.729329 9140 ts_tablet_manager.cc:1939] T 943b3ca9da9644dc9c41f1e544522453 P 44ddc6ab2946434daffc23c828accc39: Deleting consensus metadata
I20260924 17:44:12.729450 9116 tablet_bootstrap.cc:492] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Bootstrap starting.
I20260924 17:44:12.729668 8591 catalog_manager.cc:4841] TS 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661): tablet 943b3ca9da9644dc9c41f1e544522453 (table test-table [id=596669469816489e8c97090408146b28]) successfully deleted
I20260924 17:44:12.729812 9022 tablet_service.cc:1467] Processing CreateTablet for tablet c4afaaedf920406f9dcfd9c7d721e90d (DEFAULT_TABLE table=test-table [id=19b7dbe35c79449e89cba8209d546cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:12.729859 9116 tablet_bootstrap.cc:654] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.729936 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4afaaedf920406f9dcfd9c7d721e90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.730307 8742 tablet_service.cc:1467] Processing CreateTablet for tablet c4afaaedf920406f9dcfd9c7d721e90d (DEFAULT_TABLE table=test-table [id=19b7dbe35c79449e89cba8209d546cce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:12.730419 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4afaaedf920406f9dcfd9c7d721e90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.730481 9116 tablet_bootstrap.cc:492] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: No bootstrap required, opened a new log
I20260924 17:44:12.730521 9116 ts_tablet_manager.cc:1397] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:12.730686 9116 raft_consensus.cc:348] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.730753 9116 raft_consensus.cc:374] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.730772 9116 raft_consensus.cc:729] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddc6ab2946434daffc23c828accc39, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.730815 9116 consensus_queue.cc:260] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.730918 9116 ts_tablet_manager.cc:1428] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.731325 9117 tablet_bootstrap.cc:492] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Bootstrap starting.
I20260924 17:44:12.731702 8592 catalog_manager.cc:4841] TS 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699): tablet 943b3ca9da9644dc9c41f1e544522453 (table test-table [id=596669469816489e8c97090408146b28]) successfully deleted
I20260924 17:44:12.731756 9118 tablet_bootstrap.cc:492] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Bootstrap starting.
I20260924 17:44:12.732095 9117 tablet_bootstrap.cc:654] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.732196 9118 tablet_bootstrap.cc:654] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.732725 9141 tablet_replica.cc:330] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: stopping tablet replica
I20260924 17:44:12.732828 9118 tablet_bootstrap.cc:492] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: No bootstrap required, opened a new log
I20260924 17:44:12.732878 9118 ts_tablet_manager.cc:1397] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:12.732892 9141 raft_consensus.cc:2229] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:12.733019 9141 raft_consensus.cc:2258] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:12.733039 9118 raft_consensus.cc:348] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.733141 9118 raft_consensus.cc:374] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.733167 9118 raft_consensus.cc:729] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.733217 9118 consensus_queue.cc:260] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.733309 9118 ts_tablet_manager.cc:1428] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:12.733708 9117 tablet_bootstrap.cc:492] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: No bootstrap required, opened a new log
I20260924 17:44:12.733784 9117 ts_tablet_manager.cc:1397] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:44:12.733930 9117 raft_consensus.cc:348] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.733989 9141 ts_tablet_manager.cc:1905] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:12.734004 9117 raft_consensus.cc:374] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.734136 9117 raft_consensus.cc:729] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98357dba68b41b496e82d5d591b8b45, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.734195 9117 consensus_queue.cc:260] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.734284 9117 ts_tablet_manager.cc:1428] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.735895 9141 ts_tablet_manager.cc:1918] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:12.736006 9141 log.cc:1196] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/943b3ca9da9644dc9c41f1e544522453
I20260924 17:44:12.736272 9141 ts_tablet_manager.cc:1939] T 943b3ca9da9644dc9c41f1e544522453 P b98357dba68b41b496e82d5d591b8b45: Deleting consensus metadata
I20260924 17:44:12.736711 8591 catalog_manager.cc:4841] TS b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851): tablet 943b3ca9da9644dc9c41f1e544522453 (table test-table [id=596669469816489e8c97090408146b28]) successfully deleted
W20260924 17:44:12.773114 8809 tablet.cc:2369] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:44:12.795811 9126 raft_consensus.cc:482] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:12.795915 9126 raft_consensus.cc:504] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.796142 9126 leader_election.cc:288] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:12.796387 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afaaedf920406f9dcfd9c7d721e90d" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45" is_pre_election: true
I20260924 17:44:12.796399 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afaaedf920406f9dcfd9c7d721e90d" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39" is_pre_election: true
I20260924 17:44:12.796495 9042 raft_consensus.cc:2454] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 0.
I20260924 17:44:12.796521 8902 raft_consensus.cc:2454] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 0.
I20260924 17:44:12.796679 8695 leader_election.cc:302] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, b98357dba68b41b496e82d5d591b8b45; no voters:
I20260924 17:44:12.796818 9126 raft_consensus.cc:2781] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:44:12.796877 9126 raft_consensus.cc:482] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:44:12.796900 9126 raft_consensus.cc:3037] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.797809 9126 raft_consensus.cc:504] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.797973 9126 leader_election.cc:288] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 election: Requested vote from peers 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:12.798188 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afaaedf920406f9dcfd9c7d721e90d" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39"
I20260924 17:44:12.798280 8902 raft_consensus.cc:3037] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.798288 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4afaaedf920406f9dcfd9c7d721e90d" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45"
I20260924 17:44:12.798412 9042 raft_consensus.cc:3037] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.798951 8902 raft_consensus.cc:2454] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 1.
I20260924 17:44:12.799129 8696 leader_election.cc:302] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, 44ddc6ab2946434daffc23c828accc39; no voters:
I20260924 17:44:12.799242 9042 raft_consensus.cc:2454] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 1.
I20260924 17:44:12.799355 9126 raft_consensus.cc:2781] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:12.799409 9126 raft_consensus.cc:686] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 1 LEADER]: Becoming Leader. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Running, Role: LEADER
I20260924 17:44:12.799455 9126 consensus_queue.cc:237] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.800243 8604 catalog_manager.cc:5495] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2898c857d10341fe86d401c94d7ac286 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "2898c857d10341fe86d401c94d7ac286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } health_report { overall_health: UNKNOWN } } }
Sep 24 17:44:12 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271852, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.190@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:12.845466 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.831227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.1.8.190:41207)
0924 17:44:12.831291 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.831293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.831450 (+ 157us) server_negotiation.cc:407] Connection header received
0924 17:44:12.831576 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.831577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.831588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.831637 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:12.832119 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.832692 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.833402 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.833595 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.833683 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.842859 (+ 9176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:12.842860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:12.842861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:12.842869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:12.844248 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.844636 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.844637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.844638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.844696 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:12.844889 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:12.844890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:12.844890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:12.845009 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:12.845085 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.845243 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.845352 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:44:12.845541 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.831180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41207 (local address 127.0.0.1:54400)
0924 17:44:12.831354 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.831371 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.831506 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.831672 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.831673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.831987 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:12.832032 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.832034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.832775 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.832776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.833307 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.833308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.833685 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.833686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.833742 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.841211 (+ 7469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:12.841217 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:12.842746 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:12.844354 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.844362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.844365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.844559 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.844765 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:12.844766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:12.844766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:12.844827 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:12.845075 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:12.845084 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:12.845130 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.845222 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.845414 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":82,"thread_start_us":105,"threads_started":1}
I20260924 17:44:12.847749 8604 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:54400:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 17:44:12.847981 8604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:44:12.849562 8742 tablet_service.cc:1467] Processing CreateTablet for tablet 7a75f2bd87f148058d06eaa428aeffc2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d93d140fb63542698283a50498556303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:44:12.849663 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 7a75f2bd87f148058d06eaa428aeffc2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d93d140fb63542698283a50498556303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:44:12.849727 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a75f2bd87f148058d06eaa428aeffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.849824 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a75f2bd87f148058d06eaa428aeffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.851073 9118 tablet_bootstrap.cc:492] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: Bootstrap starting.
I20260924 17:44:12.851191 9117 tablet_bootstrap.cc:492] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: Bootstrap starting.
I20260924 17:44:12.851168 8878 tablet_service.cc:1467] Processing CreateTablet for tablet 7a75f2bd87f148058d06eaa428aeffc2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d93d140fb63542698283a50498556303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 17:44:12.851302 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a75f2bd87f148058d06eaa428aeffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:12.851434 9118 tablet_bootstrap.cc:654] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.851596 9117 tablet_bootstrap.cc:654] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.852219 9117 tablet_bootstrap.cc:492] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: No bootstrap required, opened a new log
I20260924 17:44:12.852272 9117 ts_tablet_manager.cc:1397] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:12.852414 9117 raft_consensus.cc:348] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.852494 9117 raft_consensus.cc:374] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.852517 9117 raft_consensus.cc:729] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98357dba68b41b496e82d5d591b8b45, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.852560 9117 consensus_queue.cc:260] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.852662 9117 ts_tablet_manager.cc:1428] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.852662 9116 tablet_bootstrap.cc:492] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: Bootstrap starting.
I20260924 17:44:12.852977 9124 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.853082 9116 tablet_bootstrap.cc:654] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:12.853123 9124 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:44:12.853821 9118 tablet_bootstrap.cc:492] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: No bootstrap required, opened a new log
I20260924 17:44:12.853859 9116 tablet_bootstrap.cc:492] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: No bootstrap required, opened a new log
I20260924 17:44:12.853894 9116 ts_tablet_manager.cc:1397] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:12.853891 9118 ts_tablet_manager.cc:1397] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 17:44:12.854038 9118 raft_consensus.cc:348] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.854108 9118 raft_consensus.cc:374] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.854123 9118 raft_consensus.cc:729] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.854065 9116 raft_consensus.cc:348] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.854177 9116 raft_consensus.cc:374] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:12.854209 9116 raft_consensus.cc:729] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddc6ab2946434daffc23c828accc39, State: Initialized, Role: FOLLOWER
I20260924 17:44:12.854174 9118 consensus_queue.cc:260] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.854259 9116 consensus_queue.cc:260] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.854286 9118 ts_tablet_manager.cc:1428] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:12.854342 9116 ts_tablet_manager.cc:1428] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:44:12.854296 9123 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.854472 9123 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:44:12.854457 9122 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.854547 9122 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 17:44:12.876284 9089 tablet.cc:2369] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:44:12.946342 9124 raft_consensus.cc:482] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:12.946511 9124 raft_consensus.cc:504] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.946909 9124 leader_election.cc:288] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661), 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699)
I20260924 17:44:12.949934 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.947065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39661 (local address 127.1.8.131:59229)
0924 17:44:12.947139 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.947153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.947222 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.947382 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.947383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.947385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.947458 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.947460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.948182 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.948183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.948995 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.948997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.949669 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.949669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.949750 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.949754 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.949821 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.949863 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 17:44:12.949934 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.947168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:59229 (local address 127.1.8.129:39661)
0924 17:44:12.947210 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.947213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.947222 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:44:12.947282 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.947283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.947292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.947366 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.947528 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.948085 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.949138 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.949586 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.949678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.949689 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.949826 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.949863 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:44:12.950188 8762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a75f2bd87f148058d06eaa428aeffc2" candidate_uuid: "b98357dba68b41b496e82d5d591b8b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2898c857d10341fe86d401c94d7ac286" is_pre_election: true
I20260924 17:44:12.950335 8762 raft_consensus.cc:2454] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98357dba68b41b496e82d5d591b8b45 in term 0.
I20260924 17:44:12.950626 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.947167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:33485 (local address 127.1.8.130:36699)
0924 17:44:12.947222 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.947224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.947301 (+ 77us) server_negotiation.cc:407] Connection header received
0924 17:44:12.947394 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.947395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.947405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.947464 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:12.947623 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.948265 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.949711 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.950169 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.950268 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.950278 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.950497 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.950546 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:44:12.950577 8975 leader_election.cc:302] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, b98357dba68b41b496e82d5d591b8b45; no voters:
I20260924 17:44:12.950649 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.947075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36699 (local address 127.1.8.131:33485)
0924 17:44:12.947217 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.947228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.947318 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.947467 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.947468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.947469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:12.947552 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.947554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.948355 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.948357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.949614 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.949617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.950271 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.950272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.950370 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.950373 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.950482 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.950522 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260924 17:44:12.950740 9124 raft_consensus.cc:2781] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:44:12.950769 9124 raft_consensus.cc:482] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:44:12.950789 9124 raft_consensus.cc:3037] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.950914 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a75f2bd87f148058d06eaa428aeffc2" candidate_uuid: "b98357dba68b41b496e82d5d591b8b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39" is_pre_election: true
I20260924 17:44:12.951013 8902 raft_consensus.cc:2454] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b98357dba68b41b496e82d5d591b8b45 in term 0.
I20260924 17:44:12.951578 9124 raft_consensus.cc:504] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.951733 9124 leader_election.cc:288] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [CANDIDATE]: Term 1 election: Requested vote from peers 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661), 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699)
I20260924 17:44:12.951889 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a75f2bd87f148058d06eaa428aeffc2" candidate_uuid: "b98357dba68b41b496e82d5d591b8b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39"
I20260924 17:44:12.951898 8762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a75f2bd87f148058d06eaa428aeffc2" candidate_uuid: "b98357dba68b41b496e82d5d591b8b45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2898c857d10341fe86d401c94d7ac286"
I20260924 17:44:12.951968 8762 raft_consensus.cc:3037] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.951968 8902 raft_consensus.cc:3037] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:12.952706 8902 raft_consensus.cc:2454] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98357dba68b41b496e82d5d591b8b45 in term 1.
I20260924 17:44:12.952811 8762 raft_consensus.cc:2454] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b98357dba68b41b496e82d5d591b8b45 in term 1.
I20260924 17:44:12.952930 8976 leader_election.cc:302] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ddc6ab2946434daffc23c828accc39, b98357dba68b41b496e82d5d591b8b45; no voters:
I20260924 17:44:12.953037 9124 raft_consensus.cc:2781] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:12.953240 9124 raft_consensus.cc:686] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [term 1 LEADER]: Becoming Leader. State: Replica: b98357dba68b41b496e82d5d591b8b45, State: Running, Role: LEADER
I20260924 17:44:12.953329 9124 consensus_queue.cc:237] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:12.953708 9156 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: TxnStatusTablet state changed. Reason: New leader b98357dba68b41b496e82d5d591b8b45. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.953799 9156 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:44:12.954027 9158 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 17:44:12.954077 8604 catalog_manager.cc:5495] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 reported cstate change: term changed from 0 to 1, leader changed from <none> to b98357dba68b41b496e82d5d591b8b45 (127.1.8.131). New cstate: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } health_report { overall_health: HEALTHY } } }
I20260924 17:44:12.954157 9158 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 17:44:12.970969 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.968210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.1.8.131:45851)
0924 17:44:12.968277 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.968279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.968290 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:44:12.968354 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.968355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.968375 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.968453 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:12.968563 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.969331 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.970063 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.970284 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.970362 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.970601 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.970721 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.970805 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.970861 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.970884 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:44:12.970971 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.968126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45851 (local address 127.0.0.1:37294)
0924 17:44:12.968216 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.968230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.968281 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.968455 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.968456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.968457 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:44:12.968504 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.968506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.969412 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.969412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.969969 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.969971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.970362 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.970363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.970465 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.970489 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.970791 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.970800 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.970855 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.970903 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 17:44:12.972066 8902 raft_consensus.cc:1261] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Refusing update from remote peer b98357dba68b41b496e82d5d591b8b45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:12.972121 8762 raft_consensus.cc:1261] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Refusing update from remote peer b98357dba68b41b496e82d5d591b8b45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:12.972304 9156 consensus_queue.cc:1035] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:12.972404 9124 consensus_queue.cc:1035] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:12.972925 9122 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: TxnStatusTablet state changed. Reason: New leader b98357dba68b41b496e82d5d591b8b45. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.973024 9122 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:44:12.973124 9123 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: TxnStatusTablet state changed. Reason: New leader b98357dba68b41b496e82d5d591b8b45. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.973253 9123 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:44:12.973948 9124 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.974032 9124 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:44:12.974076 9124 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.974108 9124 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P b98357dba68b41b496e82d5d591b8b45: This TxnStatusTablet replica's current role is: LEADER
I20260924 17:44:12.974157 9130 mvcc.cc:205] Tried to move back new op lower bound from 7332953509771476992 to 7332953509697396736. Current Snapshot: MvccSnapshot[applied={T|T < 7332953509771476992 or (T in {7332953509771476992})}]
I20260924 17:44:12.974604 9122 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.974684 9122 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P 44ddc6ab2946434daffc23c828accc39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:44:12.975258 9123 tablet_replica.cc:439] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b98357dba68b41b496e82d5d591b8b45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } } }
I20260924 17:44:12.975342 9123 tablet_replica.cc:442] T 7a75f2bd87f148058d06eaa428aeffc2 P 2898c857d10341fe86d401c94d7ac286: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 17:44:12.980428 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.977642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45851 (local address 127.1.8.129:44463)
0924 17:44:12.977787 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.977799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.977859 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.977980 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.977980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.977981 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:44:12.978024 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.978025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.978931 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.978931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.979596 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.979598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.980066 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.980067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.980106 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.980121 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.980294 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.980295 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.980329 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.980368 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260924 17:44:12.980432 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.977728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:44463 (local address 127.1.8.131:45851)
0924 17:44:12.977773 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.977776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.977849 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:44:12.977900 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.977901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.977912 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.977978 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:12.978060 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.978852 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.979688 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.979989 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.980062 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.980201 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.980236 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.980289 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.980335 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.980355 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:44:12.983116 9042 raft_consensus.cc:1261] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Refusing update from remote peer 2898c857d10341fe86d401c94d7ac286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:12.983173 8902 raft_consensus.cc:1261] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Refusing update from remote peer 2898c857d10341fe86d401c94d7ac286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:12.983348 9123 consensus_queue.cc:1035] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:12.983446 9126 consensus_queue.cc:1035] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:12.991983 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.989166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39661 (local address 127.1.8.131:32995)
0924 17:44:12.989220 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:12.989232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:12.989272 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:12.989399 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:12.989399 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:12.989401 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:44:12.989470 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.989471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.990383 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.990384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.991141 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:12.991144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.991589 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.991590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:12.991653 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.991666 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.991857 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.991858 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:44:12.991890 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 17:44:12.991924 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 17:44:12.991999 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:12.989212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:32995 (local address 127.1.8.129:39661)
0924 17:44:12.989251 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:12.989253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:12.989261 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:44:12.989329 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:12.989330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:12.989341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:12.989398 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:12.989542 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.990296 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.991212 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:12.991515 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:12.991590 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:12.991752 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:12.991788 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:12.991840 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:12.991898 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 17:44:12.991921 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:44:13.000864 8603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:13.000998 8603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:13.001648 8603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Sending DeleteTablet for 3 replicas of tablet c4afaaedf920406f9dcfd9c7d721e90d
I20260924 17:44:13.001896 8742 tablet_service.cc:1514] Processing DeleteTablet for tablet c4afaaedf920406f9dcfd9c7d721e90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:12 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260924 17:44:13.002002 8878 tablet_service.cc:1514] Processing DeleteTablet for tablet c4afaaedf920406f9dcfd9c7d721e90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:12 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33776
I20260924 17:44:13.002122 9139 tablet_replica.cc:330] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: stopping tablet replica
I20260924 17:44:13.002118 9140 tablet_replica.cc:330] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: stopping tablet replica
I20260924 17:44:13.002180 9140 raft_consensus.cc:2229] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:13.002183 9139 raft_consensus.cc:2229] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:44:13.002218 9140 raft_consensus.cc:2258] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.002244 9139 raft_consensus.cc:2258] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.002381 9022 tablet_service.cc:1514] Processing DeleteTablet for tablet c4afaaedf920406f9dcfd9c7d721e90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:12 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37284
I20260924 17:44:13.002488 9141 tablet_replica.cc:330] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: stopping tablet replica
I20260924 17:44:13.002537 9141 raft_consensus.cc:2229] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:13.002570 9141 raft_consensus.cc:2258] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.002597 9140 ts_tablet_manager.cc:1905] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.002961 9141 ts_tablet_manager.cc:1905] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.004561 9140 ts_tablet_manager.cc:1918] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:44:13.004571 9141 ts_tablet_manager.cc:1918] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:44:13.004617 9140 log.cc:1196] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/c4afaaedf920406f9dcfd9c7d721e90d
I20260924 17:44:13.004617 9141 log.cc:1196] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/c4afaaedf920406f9dcfd9c7d721e90d
I20260924 17:44:13.004841 9140 ts_tablet_manager.cc:1939] T c4afaaedf920406f9dcfd9c7d721e90d P 44ddc6ab2946434daffc23c828accc39: Deleting consensus metadata
I20260924 17:44:13.004850 9141 ts_tablet_manager.cc:1939] T c4afaaedf920406f9dcfd9c7d721e90d P b98357dba68b41b496e82d5d591b8b45: Deleting consensus metadata
I20260924 17:44:13.005220 8592 catalog_manager.cc:4841] TS 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699): tablet c4afaaedf920406f9dcfd9c7d721e90d (table test-table [id=19b7dbe35c79449e89cba8209d546cce]) successfully deleted
I20260924 17:44:13.005491 9139 ts_tablet_manager.cc:1905] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.005679 8591 catalog_manager.cc:4841] TS b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851): tablet c4afaaedf920406f9dcfd9c7d721e90d (table test-table [id=19b7dbe35c79449e89cba8209d546cce]) successfully deleted
I20260924 17:44:13.007346 9139 ts_tablet_manager.cc:1918] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:44:13.007416 9139 log.cc:1196] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/c4afaaedf920406f9dcfd9c7d721e90d
I20260924 17:44:13.007619 9139 ts_tablet_manager.cc:1939] T c4afaaedf920406f9dcfd9c7d721e90d P 2898c857d10341fe86d401c94d7ac286: Deleting consensus metadata
I20260924 17:44:13.008273 8591 catalog_manager.cc:4841] TS 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661): tablet c4afaaedf920406f9dcfd9c7d721e90d (table test-table [id=19b7dbe35c79449e89cba8209d546cce]) successfully deleted
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[8560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:44:13.027263 8665 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:13.021233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.1.8.190:41207)
0924 17:44:13.021306 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.021309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.021464 (+ 155us) server_negotiation.cc:407] Connection header received
0924 17:44:13.021514 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.021515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.021527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.021604 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:13.022154 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.022977 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.023743 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.023944 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.024022 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.027174 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54404: BlockingRecv error: recv got EOF from 127.0.0.1:54404 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[8560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM
I20260924 17:44:13.036067 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.028161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.1.8.190:41207)
0924 17:44:13.028204 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.028206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.028283 (+ 77us) server_negotiation.cc:407] Connection header received
0924 17:44:13.028348 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.028349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.028358 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.028404 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:13.028979 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.029460 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.030503 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.030708 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.030918 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.033342 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:13.033343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:13.033344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:13.033352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:13.034875 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:13.035230 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:13.035232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:13.035232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:13.035309 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:13.035521 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:13.035522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:13.035522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:13.035647 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:13.035755 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.035859 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.035948 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:44:13.038012 8603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:44:13.038203 8603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:44:13.039713 8878 tablet_service.cc:1467] Processing CreateTablet for tablet 087313e898f447ccb16f518b8081cfbf (DEFAULT_TABLE table=test-table [id=0954e00e5143472392b5ce877c9cedcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:13.039729 9022 tablet_service.cc:1467] Processing CreateTablet for tablet 087313e898f447ccb16f518b8081cfbf (DEFAULT_TABLE table=test-table [id=0954e00e5143472392b5ce877c9cedcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:13.039731 8742 tablet_service.cc:1467] Processing CreateTablet for tablet 087313e898f447ccb16f518b8081cfbf (DEFAULT_TABLE table=test-table [id=0954e00e5143472392b5ce877c9cedcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:13.039868 8742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087313e898f447ccb16f518b8081cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:13.039867 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087313e898f447ccb16f518b8081cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:13.039868 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 087313e898f447ccb16f518b8081cfbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:13.041210 9118 tablet_bootstrap.cc:492] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Bootstrap starting.
I20260924 17:44:13.041250 9117 tablet_bootstrap.cc:492] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Bootstrap starting.
I20260924 17:44:13.041517 9118 tablet_bootstrap.cc:654] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:13.041631 9117 tablet_bootstrap.cc:654] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:13.041965 9116 tablet_bootstrap.cc:492] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Bootstrap starting.
I20260924 17:44:13.042086 9118 tablet_bootstrap.cc:492] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: No bootstrap required, opened a new log
I20260924 17:44:13.042129 9118 ts_tablet_manager.cc:1397] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:13.042227 9118 raft_consensus.cc:348] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.042275 9118 raft_consensus.cc:374] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:13.042289 9118 raft_consensus.cc:729] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Initialized, Role: FOLLOWER
I20260924 17:44:13.042341 9116 tablet_bootstrap.cc:654] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:13.042347 9118 consensus_queue.cc:260] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.042577 9118 ts_tablet_manager.cc:1428] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 17:44:13.043256 9117 tablet_bootstrap.cc:492] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: No bootstrap required, opened a new log
I20260924 17:44:13.043306 9117 ts_tablet_manager.cc:1397] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:44:13.043485 9117 raft_consensus.cc:348] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.043555 9117 raft_consensus.cc:374] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:13.043576 9117 raft_consensus.cc:729] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98357dba68b41b496e82d5d591b8b45, State: Initialized, Role: FOLLOWER
I20260924 17:44:13.043627 9117 consensus_queue.cc:260] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.043732 9117 ts_tablet_manager.cc:1428] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:13.044271 9116 tablet_bootstrap.cc:492] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: No bootstrap required, opened a new log
I20260924 17:44:13.044348 9116 ts_tablet_manager.cc:1397] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:44:13.044520 9116 raft_consensus.cc:348] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.044602 9116 raft_consensus.cc:374] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:13.044642 9116 raft_consensus.cc:729] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddc6ab2946434daffc23c828accc39, State: Initialized, Role: FOLLOWER
I20260924 17:44:13.044696 9116 consensus_queue.cc:260] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.044796 9116 ts_tablet_manager.cc:1428] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:13.045584 9123 raft_consensus.cc:482] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:13.045647 9123 raft_consensus.cc:504] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.045784 9123 leader_election.cc:288] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:13.045951 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087313e898f447ccb16f518b8081cfbf" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45" is_pre_election: true
I20260924 17:44:13.045964 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087313e898f447ccb16f518b8081cfbf" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39" is_pre_election: true
I20260924 17:44:13.046038 9042 raft_consensus.cc:2454] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 0.
I20260924 17:44:13.046056 8902 raft_consensus.cc:2454] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 0.
I20260924 17:44:13.046191 8696 leader_election.cc:302] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, 44ddc6ab2946434daffc23c828accc39; no voters:
I20260924 17:44:13.046335 9123 raft_consensus.cc:2781] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:44:13.046375 9123 raft_consensus.cc:482] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:44:13.046446 9123 raft_consensus.cc:3037] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:13.047242 9123 raft_consensus.cc:504] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.047365 9123 leader_election.cc:288] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 election: Requested vote from peers 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:13.047580 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087313e898f447ccb16f518b8081cfbf" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45"
I20260924 17:44:13.047574 8902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "087313e898f447ccb16f518b8081cfbf" candidate_uuid: "2898c857d10341fe86d401c94d7ac286" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44ddc6ab2946434daffc23c828accc39"
I20260924 17:44:13.047643 9042 raft_consensus.cc:3037] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:13.047646 8902 raft_consensus.cc:3037] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:13.048369 8902 raft_consensus.cc:2454] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 1.
I20260924 17:44:13.048444 9042 raft_consensus.cc:2454] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2898c857d10341fe86d401c94d7ac286 in term 1.
I20260924 17:44:13.048542 8696 leader_election.cc:302] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2898c857d10341fe86d401c94d7ac286, 44ddc6ab2946434daffc23c828accc39; no voters:
I20260924 17:44:13.048616 9123 raft_consensus.cc:2781] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:13.048656 9123 raft_consensus.cc:686] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 1 LEADER]: Becoming Leader. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Running, Role: LEADER
I20260924 17:44:13.048696 9123 consensus_queue.cc:237] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.049247 8603 catalog_manager.cc:5495] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2898c857d10341fe86d401c94d7ac286 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "2898c857d10341fe86d401c94d7ac286" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } health_report { overall_health: UNKNOWN } } }
I20260924 17:44:13.055879 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.053237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.1.8.129:39661)
0924 17:44:13.053280 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.053282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.053293 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:44:13.053320 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.053321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.053334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.053405 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:13.053541 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.054245 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.055123 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.055426 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.055507 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.055647 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:13.055683 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:13.055735 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.055782 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.055803 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:44:13.056751 9042 raft_consensus.cc:1261] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Refusing update from remote peer 2898c857d10341fe86d401c94d7ac286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:13.056965 8902 raft_consensus.cc:1261] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Refusing update from remote peer 2898c857d10341fe86d401c94d7ac286: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:13.057174 9126 consensus_queue.cc:1035] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:13.057302 9123 consensus_queue.cc:1035] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:13.059492 8604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:13.059585 8604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:13.060441 8604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Sending DeleteTablet for 3 replicas of tablet 087313e898f447ccb16f518b8081cfbf
I20260924 17:44:13.060683 9022 tablet_service.cc:1514] Processing DeleteTablet for tablet 087313e898f447ccb16f518b8081cfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37284
I20260924 17:44:13.060714 8742 tablet_service.cc:1514] Processing DeleteTablet for tablet 087313e898f447ccb16f518b8081cfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260924 17:44:13.060755 8741 tablet_service.cc:1514] Processing DeleteTablet for tablet 087313e898f447ccb16f518b8081cfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260924 17:44:13.060829 9141 tablet_replica.cc:330] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: stopping tablet replica
I20260924 17:44:13.060899 9141 raft_consensus.cc:2229] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:13.060954 9141 raft_consensus.cc:2258] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.061028 9139 tablet_replica.cc:330] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: stopping tablet replica
I20260924 17:44:13.061076 9139 raft_consensus.cc:2229] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:44:13.061133 9139 raft_consensus.cc:2258] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.061094 8878 tablet_service.cc:1514] Processing DeleteTablet for tablet 087313e898f447ccb16f518b8081cfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33776
I20260924 17:44:13.061096 8882 tablet_service.cc:1514] Processing DeleteTablet for tablet 087313e898f447ccb16f518b8081cfbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33776
I20260924 17:44:13.061305 9140 tablet_replica.cc:330] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: stopping tablet replica
I20260924 17:44:13.061363 9140 raft_consensus.cc:2229] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:13.061419 9140 raft_consensus.cc:2258] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.061568 8605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 17:44:13.061699 9139 ts_tablet_manager.cc:1905] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:44:13.061721 8605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:44:13.061757 9140 ts_tablet_manager.cc:1905] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.063390 8882 tablet_service.cc:1467] Processing CreateTablet for tablet d3bf048b659f40aa9784361ca6390904 (DEFAULT_TABLE table=test-table [id=5b84169cae83445c9141bf831fd5c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:13.063539 9022 tablet_service.cc:1467] Processing CreateTablet for tablet d3bf048b659f40aa9784361ca6390904 (DEFAULT_TABLE table=test-table [id=5b84169cae83445c9141bf831fd5c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:13.063586 8741 tablet_service.cc:1467] Processing CreateTablet for tablet d3bf048b659f40aa9784361ca6390904 (DEFAULT_TABLE table=test-table [id=5b84169cae83445c9141bf831fd5c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:13.063599 9140 ts_tablet_manager.cc:1918] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:13.063652 9140 log.cc:1196] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/087313e898f447ccb16f518b8081cfbf
I20260924 17:44:13.063656 9022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3bf048b659f40aa9784361ca6390904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:13.063736 9139 ts_tablet_manager.cc:1918] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:13.063781 9139 log.cc:1196] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/087313e898f447ccb16f518b8081cfbf
I20260924 17:44:13.063923 9140 ts_tablet_manager.cc:1939] T 087313e898f447ccb16f518b8081cfbf P 44ddc6ab2946434daffc23c828accc39: Deleting consensus metadata
I20260924 17:44:13.063992 9139 ts_tablet_manager.cc:1939] T 087313e898f447ccb16f518b8081cfbf P 2898c857d10341fe86d401c94d7ac286: Deleting consensus metadata
I20260924 17:44:13.064081 8882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3bf048b659f40aa9784361ca6390904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:13.064126 8741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3bf048b659f40aa9784361ca6390904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:13.065439 9118 tablet_bootstrap.cc:492] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286: Bootstrap starting.
I20260924 17:44:13.065717 9118 tablet_bootstrap.cc:654] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:13.065732 9117 tablet_bootstrap.cc:492] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Bootstrap starting.
I20260924 17:44:13.065738 9116 tablet_bootstrap.cc:492] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Bootstrap starting.
I20260924 17:44:13.066119 9116 tablet_bootstrap.cc:654] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:13.066422 8591 catalog_manager.cc:4841] TS 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661): tablet 087313e898f447ccb16f518b8081cfbf (table test-table [id=0954e00e5143472392b5ce877c9cedcd]) successfully deleted
W20260924 17:44:13.066565 8591 catalog_manager.cc:4808] TS 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661): delete failed for tablet 087313e898f447ccb16f518b8081cfbf because the tablet was not found. No further retry: Not found: Tablet not found: 087313e898f447ccb16f518b8081cfbf
I20260924 17:44:13.066737 9116 tablet_bootstrap.cc:492] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: No bootstrap required, opened a new log
I20260924 17:44:13.066780 9116 ts_tablet_manager.cc:1397] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:13.066929 9116 raft_consensus.cc:348] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.067004 9116 raft_consensus.cc:374] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:13.067008 9118 tablet_bootstrap.cc:492] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286: No bootstrap required, opened a new log
I20260924 17:44:13.067029 9116 raft_consensus.cc:729] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44ddc6ab2946434daffc23c828accc39, State: Initialized, Role: FOLLOWER
I20260924 17:44:13.067045 9118 ts_tablet_manager.cc:1397] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:44:13.067097 9116 consensus_queue.cc:260] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.067186 9116 ts_tablet_manager.cc:1428] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:13.067160 9118 raft_consensus.cc:348] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.067222 9118 raft_consensus.cc:374] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:13.067245 9118 raft_consensus.cc:729] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2898c857d10341fe86d401c94d7ac286, State: Initialized, Role: FOLLOWER
I20260924 17:44:13.067284 9118 consensus_queue.cc:260] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.067365 9118 ts_tablet_manager.cc:1428] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:13.067384 9117 tablet_bootstrap.cc:654] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:13.068266 8592 catalog_manager.cc:4841] TS 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699): tablet 087313e898f447ccb16f518b8081cfbf (table test-table [id=0954e00e5143472392b5ce877c9cedcd]) successfully deleted
W20260924 17:44:13.068347 8592 catalog_manager.cc:4808] TS 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699): delete failed for tablet 087313e898f447ccb16f518b8081cfbf because the tablet was not found. No further retry: Not found: Tablet not found: 087313e898f447ccb16f518b8081cfbf
I20260924 17:44:13.068610 9141 ts_tablet_manager.cc:1905] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.068878 9117 tablet_bootstrap.cc:492] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: No bootstrap required, opened a new log
I20260924 17:44:13.068928 9117 ts_tablet_manager.cc:1397] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 17:44:13.069058 9117 raft_consensus.cc:348] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.069120 9117 raft_consensus.cc:374] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:13.069139 9117 raft_consensus.cc:729] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98357dba68b41b496e82d5d591b8b45, State: Initialized, Role: FOLLOWER
I20260924 17:44:13.069192 9117 consensus_queue.cc:260] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.069271 9117 ts_tablet_manager.cc:1428] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:13.069967 9141 ts_tablet_manager.cc:1918] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:13.070026 9141 log.cc:1196] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/087313e898f447ccb16f518b8081cfbf
I20260924 17:44:13.070262 9141 ts_tablet_manager.cc:1939] T 087313e898f447ccb16f518b8081cfbf P b98357dba68b41b496e82d5d591b8b45: Deleting consensus metadata
I20260924 17:44:13.070556 8591 catalog_manager.cc:4841] TS b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851): tablet 087313e898f447ccb16f518b8081cfbf (table test-table [id=0954e00e5143472392b5ce877c9cedcd]) successfully deleted
I20260924 17:44:13.151294 9122 raft_consensus.cc:482] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:13.151407 9122 raft_consensus.cc:504] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.151849 9122 leader_election.cc:288] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:13.155375 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.152500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:57689 (local address 127.1.8.131:45851)
0924 17:44:13.152558 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.152561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.152603 (+ 42us) server_negotiation.cc:407] Connection header received
0924 17:44:13.152730 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.152731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.152740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.152799 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:13.152986 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.153541 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.154643 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.155014 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.155077 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.155085 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.155260 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.155301 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:44:13.155416 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.152314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:40401 (local address 127.1.8.129:39661)
0924 17:44:13.152389 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.152407 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.152436 (+ 29us) server_negotiation.cc:407] Connection header received
0924 17:44:13.152626 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.152627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.152639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.152700 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:13.152911 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.153456 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.154693 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.155034 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.155133 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.155143 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.155306 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.155351 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:44:13.155414 9194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.152414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45851 (local address 127.1.8.130:57689)
0924 17:44:13.152514 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:13.152525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:13.152620 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:13.152821 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:13.152821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:13.152823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:13.152912 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.152913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.153657 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.153658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.154560 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.154563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.155107 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.155109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.155185 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.155190 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:44:13.155233 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 17:44:13.155282 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":34,"thread_start_us":45,"threads_started":1}
I20260924 17:44:13.155694 9193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.152201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39661 (local address 127.1.8.130:40401)
0924 17:44:13.152338 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:13.152352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:13.152450 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:13.152728 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:13.152729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:13.152731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:13.152831 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.152834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.153604 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.153605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.154587 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.154589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.155145 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.155146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.155211 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.155214 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:13.155289 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 17:44:13.155327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":49,"thread_start_us":85,"threads_started":1}
I20260924 17:44:13.155696 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bf048b659f40aa9784361ca6390904" candidate_uuid: "44ddc6ab2946434daffc23c828accc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45" is_pre_election: true
I20260924 17:44:13.155789 9042 raft_consensus.cc:2454] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44ddc6ab2946434daffc23c828accc39 in term 0.
I20260924 17:44:13.156023 8835 leader_election.cc:302] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ddc6ab2946434daffc23c828accc39, b98357dba68b41b496e82d5d591b8b45; no voters:
I20260924 17:44:13.156039 8762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bf048b659f40aa9784361ca6390904" candidate_uuid: "44ddc6ab2946434daffc23c828accc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2898c857d10341fe86d401c94d7ac286" is_pre_election: true
I20260924 17:44:13.156124 8762 raft_consensus.cc:2454] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44ddc6ab2946434daffc23c828accc39 in term 0.
I20260924 17:44:13.156172 9122 raft_consensus.cc:2781] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:44:13.156216 9122 raft_consensus.cc:482] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:44:13.156240 9122 raft_consensus.cc:3037] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:13.156948 9122 raft_consensus.cc:504] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.157100 9122 leader_election.cc:288] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [CANDIDATE]: Term 1 election: Requested vote from peers 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661), b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851)
I20260924 17:44:13.157305 8762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bf048b659f40aa9784361ca6390904" candidate_uuid: "44ddc6ab2946434daffc23c828accc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2898c857d10341fe86d401c94d7ac286"
I20260924 17:44:13.157325 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bf048b659f40aa9784361ca6390904" candidate_uuid: "44ddc6ab2946434daffc23c828accc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b98357dba68b41b496e82d5d591b8b45"
I20260924 17:44:13.157393 9042 raft_consensus.cc:3037] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:13.157406 8762 raft_consensus.cc:3037] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:13.158022 9042 raft_consensus.cc:2454] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44ddc6ab2946434daffc23c828accc39 in term 1.
I20260924 17:44:13.158138 8762 raft_consensus.cc:2454] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44ddc6ab2946434daffc23c828accc39 in term 1.
I20260924 17:44:13.158167 8835 leader_election.cc:302] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44ddc6ab2946434daffc23c828accc39, b98357dba68b41b496e82d5d591b8b45; no voters:
I20260924 17:44:13.158344 9122 raft_consensus.cc:2781] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:13.158502 9122 raft_consensus.cc:686] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 1 LEADER]: Becoming Leader. State: Replica: 44ddc6ab2946434daffc23c828accc39, State: Running, Role: LEADER
I20260924 17:44:13.158566 9122 consensus_queue.cc:237] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } }
I20260924 17:44:13.159381 8605 catalog_manager.cc:5495] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44ddc6ab2946434daffc23c828accc39 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "44ddc6ab2946434daffc23c828accc39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44ddc6ab2946434daffc23c828accc39" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 36699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 } health_report { overall_health: UNKNOWN } } }
I20260924 17:44:13.187824 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.185310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.1.8.130:36699)
0924 17:44:13.185381 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.185383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.185392 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:44:13.185436 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.185437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.185459 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.185545 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:13.185665 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.186161 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.187027 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.187231 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.187304 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.187468 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:13.187583 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:13.187679 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.187718 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.187737 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:44:13.192245 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.190087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:58131 (local address 127.1.8.131:45851)
0924 17:44:13.190148 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.190150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.190238 (+ 88us) server_negotiation.cc:407] Connection header received
0924 17:44:13.190286 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.190287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.190296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.190340 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:13.190498 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.190937 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.191634 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.191857 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.191916 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.192055 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:13.192079 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:13.192117 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.192172 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.192186 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:44:13.192276 9199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.190032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45851 (local address 127.1.8.130:58131)
0924 17:44:13.190172 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:13.190182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:13.190244 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:13.190347 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:13.190348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:13.190350 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:44:13.190447 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.190448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.190996 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.190997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.191558 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.191559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.191928 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.191928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.191973 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.191986 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:44:13.192129 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:44:13.192131 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:44:13.192162 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 17:44:13.192196 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260924 17:44:13.194514 9042 raft_consensus.cc:1261] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Refusing update from remote peer 44ddc6ab2946434daffc23c828accc39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:13.194540 8762 raft_consensus.cc:1261] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Refusing update from remote peer 44ddc6ab2946434daffc23c828accc39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:13.194792 9122 consensus_queue.cc:1035] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2898c857d10341fe86d401c94d7ac286" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:13.194970 9195 consensus_queue.cc:1035] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b98357dba68b41b496e82d5d591b8b45" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:13.201462 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.199307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:49075 (local address 127.1.8.130:36699)
0924 17:44:13.199345 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:13.199347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:13.199355 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:44:13.199419 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:13.199419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:13.199427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:13.199499 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:13.199659 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.200198 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.200882 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.201056 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.201106 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.201210 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:13.201234 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:13.201265 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:13.201335 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 17:44:13.201374 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:44:13.201474 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.199252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:36699 (local address 127.1.8.131:49075)
0924 17:44:13.199305 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:13.199317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:13.199354 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:13.199519 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:13.199520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:13.199522 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 17:44:13.199569 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.199571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.200268 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.200268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.200807 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:13.200808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:13.201118 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:13.201119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:13.201155 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:13.201159 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 17:44:13.201266 (+ 107us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 17:44:13.201267 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 17:44:13.201333 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:44:13.201377 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 17:44:13.208634 8605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:13.208742 8605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:13.209394 8605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6881a2b7d5b4585aeccd2c54b62a309: Sending DeleteTablet for 3 replicas of tablet d3bf048b659f40aa9784361ca6390904
I20260924 17:44:13.209664 9022 tablet_service.cc:1514] Processing DeleteTablet for tablet d3bf048b659f40aa9784361ca6390904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:37284
I20260924 17:44:13.209674 8741 tablet_service.cc:1514] Processing DeleteTablet for tablet d3bf048b659f40aa9784361ca6390904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42350
I20260924 17:44:13.209793 9141 tablet_replica.cc:330] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: stopping tablet replica
I20260924 17:44:13.209811 8882 tablet_service.cc:1514] Processing DeleteTablet for tablet d3bf048b659f40aa9784361ca6390904 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:13 UTC) from {username='oryx', principal='oryx/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33776
I20260924 17:44:13.209856 9141 raft_consensus.cc:2229] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:13.209887 9140 tablet_replica.cc:330] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: stopping tablet replica
I20260924 17:44:13.209906 9141 raft_consensus.cc:2258] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.209925 9140 raft_consensus.cc:2229] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:44:13.209976 9139 tablet_replica.cc:330] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286: stopping tablet replica
I20260924 17:44:13.210011 9140 raft_consensus.cc:2258] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.210035 9139 raft_consensus.cc:2229] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:13.210081 9139 raft_consensus.cc:2258] T d3bf048b659f40aa9784361ca6390904 P 2898c857d10341fe86d401c94d7ac286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:13.210454 9140 ts_tablet_manager.cc:1905] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.210530 9141 ts_tablet_manager.cc:1905] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:13.211862 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 8677
I20260924 17:44:13.213671 9140 ts_tablet_manager.cc:1918] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:44:13.213765 9140 log.cc:1196] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/d3bf048b659f40aa9784361ca6390904
I20260924 17:44:13.213975 9140 ts_tablet_manager.cc:1939] T d3bf048b659f40aa9784361ca6390904 P 44ddc6ab2946434daffc23c828accc39: Deleting consensus metadata
I20260924 17:44:13.214335 9141 ts_tablet_manager.cc:1918] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 17:44:13.214344 8592 catalog_manager.cc:4841] TS 44ddc6ab2946434daffc23c828accc39 (127.1.8.130:36699): tablet d3bf048b659f40aa9784361ca6390904 (table test-table [id=5b84169cae83445c9141bf831fd5c51c]) successfully deleted
I20260924 17:44:13.214426 9141 log.cc:1196] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/d3bf048b659f40aa9784361ca6390904
I20260924 17:44:13.214632 9141 ts_tablet_manager.cc:1939] T d3bf048b659f40aa9784361ca6390904 P b98357dba68b41b496e82d5d591b8b45: Deleting consensus metadata
I20260924 17:44:13.215024 8591 catalog_manager.cc:4841] TS b98357dba68b41b496e82d5d591b8b45 (127.1.8.131:45851): tablet d3bf048b659f40aa9784361ca6390904 (table test-table [id=5b84169cae83445c9141bf831fd5c51c]) successfully deleted
W20260924 17:44:13.218603 8591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:39661 (error 108)
I20260924 17:44:13.218832 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 8817
I20260924 17:44:13.219620 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:13.218952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39661 (local address 127.0.0.1:42372)
0924 17:44:13.219537 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:13.219579 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:39661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":33,"threads_started":1}
W20260924 17:44:13.219713 8591 catalog_manager.cc:4584] TS 2898c857d10341fe86d401c94d7ac286 (127.1.8.129:39661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3bf048b659f40aa9784361ca6390904: Network error: Client connection negotiation failed: client connection to 127.1.8.129:39661: connect: Connection refused (error 111)
I20260924 17:44:13.223645 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 8957
I20260924 17:44:13.229470 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 8573
2026-09-24T17:44:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2464 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[9209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[9209](info): set up 2 sockets
Sep 24 17:44:13 dist-test-slave-r7hf krb5kdc[9209](info): commencing operation
krb5kdc: starting...
W20260924 17:44:14.275050 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:44:14 dist-test-slave-r7hf krb5kdc[9209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:44:14.405076 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:32893
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:32893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:14.493602 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:14.493955 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:14.493997 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:14.496143 9222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:14.496209 9222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:14.496235 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:14.496266 9222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:14.496294 9222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:14.498498 9222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:32893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:32893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:14.498808 9222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:14.499083 9222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:14.504439 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:14.504439 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:14.504644 9222 server_base.cc:1034] running on GCE node
W20260924 17:44:14.504781 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:14.505045 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:14.505414 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:14.506590 9222 hybrid_clock.cc:648] HybridClock initialized: now 1790271854506574 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T17:44:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1281 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:44:14 dist-test-slave-r7hf krb5kdc[9235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:44:14 dist-test-slave-r7hf krb5kdc[9235](info): set up 2 sockets
Sep 24 17:44:14 dist-test-slave-r7hf krb5kdc[9235](info): commencing operation
krb5kdc: starting...
W20260924 17:44:15.554929 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:44:15 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:44:15.681984 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39247
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:15.766749 9248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:15.767015 9248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:15.767055 9248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:15.768253 9248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:15.768330 9248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:15.768354 9248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:15.768375 9248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:15.768406 9248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:15.769933 9248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:15.770236 9248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:15.770498 9248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:15.775588 9256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:15.775601 9253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:15.775601 9254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:15.775619 9248 server_base.cc:1034] running on GCE node
I20260924 17:44:15.776080 9248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:15.776381 9248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:15.777557 9248 hybrid_clock.cc:648] HybridClock initialized: now 1790271855777533 us; error 48 us; skew 500 ppm
Sep 24 17:44:15 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:15.780138 9248 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:44:15.780682 9248 webserver.cc:460] Webserver started at http://127.1.8.190:42611/ using document root <none> and password file <none>
I20260924 17:44:15.780927 9248 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:15.780997 9248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:15.781183 9248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:15.782281 9248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "dffe8d883a1f4425b43622fd522bbd37"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "6cfdf9648926e948feadca4a0ff76ae9"
server_key_iv: "47a493c144ed88dabeb6c7b40218bf45"
server_key_version: "encryptionkey@0"
I20260924 17:44:15.782734 9248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "dffe8d883a1f4425b43622fd522bbd37"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "6cfdf9648926e948feadca4a0ff76ae9"
server_key_iv: "47a493c144ed88dabeb6c7b40218bf45"
server_key_version: "encryptionkey@0"
I20260924 17:44:15.784094 9248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:15.784806 9263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:15.784978 9248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:15.785089 9248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "dffe8d883a1f4425b43622fd522bbd37"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "6cfdf9648926e948feadca4a0ff76ae9"
server_key_iv: "47a493c144ed88dabeb6c7b40218bf45"
server_key_version: "encryptionkey@0"
I20260924 17:44:15.785171 9248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:15.803130 9248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:15.805315 9248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:15.805491 9248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:15.809909 9248 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39247
I20260924 17:44:15.809957 9315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39247 every 8 connection(s)
I20260924 17:44:15.810357 9248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:44:15.811303 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:15.813668 9316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Bootstrap starting.
I20260924 17:44:15.814270 9316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:15.814617 9316 log.cc:824] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:15.815330 9316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: No bootstrap required, opened a new log
I20260924 17:44:15.816956 9316 raft_consensus.cc:348] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:15.817137 9316 raft_consensus.cc:374] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:15.817163 9316 raft_consensus.cc:729] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffe8d883a1f4425b43622fd522bbd37, State: Initialized, Role: FOLLOWER
I20260924 17:44:15.817349 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9248
I20260924 17:44:15.817333 9316 consensus_queue.cc:260] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:15.817406 9316 raft_consensus.cc:388] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:44:15.817459 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:44:15.817466 9316 raft_consensus.cc:482] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:44:15.817508 9316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:15.817665 1058 external_mini_cluster.cc:1428] Setting key 46d7d34ea30cc362d487e06025dd40c3
I20260924 17:44:15.818316 9316 raft_consensus.cc:504] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:15.818471 9316 leader_election.cc:302] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dffe8d883a1f4425b43622fd522bbd37; no voters:
I20260924 17:44:15.818640 9316 leader_election.cc:288] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:44:15.818846 9321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:15.818894 9316 sys_catalog.cc:564] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:44:15.818965 9321 raft_consensus.cc:686] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 LEADER]: Becoming Leader. State: Replica: dffe8d883a1f4425b43622fd522bbd37, State: Running, Role: LEADER
I20260924 17:44:15.819041 9321 consensus_queue.cc:237] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:15.819366 9321 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dffe8d883a1f4425b43622fd522bbd37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } } }
I20260924 17:44:15.819428 9321 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:15.819541 9321 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffe8d883a1f4425b43622fd522bbd37. Latest consensus state: current_term: 1 leader_uuid: "dffe8d883a1f4425b43622fd522bbd37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } } }
I20260924 17:44:15.819592 9321 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:15.819710 9328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:44:15.820293 9328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:44:15.821940 9328 catalog_manager.cc:1347] Generated new cluster ID: 7794f67175374f90a1cec6f66ab01325
I20260924 17:44:15.821995 9328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:44:15 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:44:15.829998 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:15.818204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.1.8.190:39247)
0924 17:44:15.818452 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:15.818462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:15.818508 (+ 46us) server_negotiation.cc:407] Connection header received
0924 17:44:15.818637 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:15.818644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:15.818710 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:15.818859 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:15.819885 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:15.821434 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:15.822501 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:15.823095 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:15.823167 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:15.826174 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:15.826178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:15.826179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:15.826192 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:15.828264 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:15.828695 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:15.828699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:15.828699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:15.828807 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:15.829030 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:15.829031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:15.829031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:15.829350 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:15.829496 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:15.829612 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 17:44:15.829745 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260924 17:44:15.848556 9328 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:44:15.849325 9328 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:44:15.856174 9328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Generated new TSK 0
I20260924 17:44:15.856437 9328 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:44:15.889916 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39247
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:15.973241 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:15.973501 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:15.973542 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:15.974769 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:15.974846 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:15.974890 9342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:44:15.976380 9342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:15.976677 9342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:15.976913 9342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:15.982172 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:15.982172 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:15.982231 9350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:15.982378 9342 server_base.cc:1034] running on GCE node
I20260924 17:44:15.982772 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:15.983071 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:15.984249 9342 hybrid_clock.cc:648] HybridClock initialized: now 1790271855984227 us; error 36 us; skew 500 ppm
Sep 24 17:44:15 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:15.986771 9342 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:44:15.987262 9342 webserver.cc:460] Webserver started at http://127.1.8.129:46747/ using document root <none> and password file <none>
I20260924 17:44:15.987501 9342 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:15.987548 9342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:15.987684 9342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:15.988842 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "fca575bd5e0e49b293be963222b4c64c"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "e8114423c3de0848aed29a68ca68962c"
server_key_iv: "cafae6672fb44655b1d6b444d29be597"
server_key_version: "encryptionkey@0"
I20260924 17:44:15.989241 9342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "fca575bd5e0e49b293be963222b4c64c"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "e8114423c3de0848aed29a68ca68962c"
server_key_iv: "cafae6672fb44655b1d6b444d29be597"
server_key_version: "encryptionkey@0"
I20260924 17:44:15.990603 9342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:44:15.991361 9357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:15.991581 9342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:15.991701 9342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "fca575bd5e0e49b293be963222b4c64c"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "e8114423c3de0848aed29a68ca68962c"
server_key_iv: "cafae6672fb44655b1d6b444d29be597"
server_key_version: "encryptionkey@0"
I20260924 17:44:15.991782 9342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.011806 9342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.013782 9342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.013959 9342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.014205 9342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:16.014590 9342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:16.014647 9342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.014689 9342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:16.014719 9342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.021131 9342 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42857
I20260924 17:44:16.021207 9470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42857 every 8 connection(s)
I20260924 17:44:16.021555 9342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:44:16.025610 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9342
I20260924 17:44:16.025707 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:44:16.025918 1058 external_mini_cluster.cc:1428] Setting key c23b6e09e9f4226284f8b042e042bc06
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271855, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:44:16.032060 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.022327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56661 (local address 127.1.8.190:39247)
0924 17:44:16.022415 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.022417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.022661 (+ 244us) server_negotiation.cc:407] Connection header received
0924 17:44:16.023216 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.023217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.023233 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.023290 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.024290 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.024799 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.025878 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.026221 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.026277 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.028714 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:16.028715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:16.028716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:16.028727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:16.030605 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.031052 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.031054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.031054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.031112 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.031378 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.031379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.031379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.031511 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:16.031583 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:16.031879 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 17:44:16.031964 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:44:16.032274 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.022318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:56661)
0924 17:44:16.022520 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.022558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.023104 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.023360 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.023364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.023986 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.024180 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.024185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.024901 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.024902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.025775 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.025779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.026295 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.026296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.026468 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.027072 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.027081 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.028605 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:16.030702 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:16.030709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:16.030711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:16.030956 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:16.031196 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:16.031197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:16.031197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:16.031311 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:16.031585 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:16.031593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:16.031738 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 17:44:16.031843 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 17:44:16.032019 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260924 17:44:16.032784 9471 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39247
I20260924 17:44:16.032897 9471 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:16.033079 9471 heartbeater.cc:511] Master 127.1.8.190:39247 requested a full tablet report, sending...
I20260924 17:44:16.033643 9280 ts_manager.cc:194] Registered new tserver with Master: fca575bd5e0e49b293be963222b4c64c (127.1.8.129:42857)
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:44:16.034313 9280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:56661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:44:16.053668 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39247
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:16.135891 9476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:16.136111 9476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:16.136134 9476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:16.137331 9476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:16.137375 9476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:16.137403 9476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:44:16.138885 9476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:16.139146 9476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:16.139382 9476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:16.144460 9482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.144435 9481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.144425 9484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.145052 9476 server_base.cc:1034] running on GCE node
I20260924 17:44:16.145291 9476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:16.145490 9476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:16.146636 9476 hybrid_clock.cc:648] HybridClock initialized: now 1790271856146624 us; error 29 us; skew 500 ppm
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:16.149251 9476 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:44:16.149665 9476 webserver.cc:460] Webserver started at http://127.1.8.130:43191/ using document root <none> and password file <none>
I20260924 17:44:16.149849 9476 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:16.149888 9476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:16.149973 9476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:16.151064 9476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "eebe69e2d97a44deadeb9e43db4d5bdd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "bd0150a3436fca8e8f70ef80d8c6c371"
server_key_iv: "4753a90f2f9c5c037b849ef3f127d052"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.151369 9476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "eebe69e2d97a44deadeb9e43db4d5bdd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "bd0150a3436fca8e8f70ef80d8c6c371"
server_key_iv: "4753a90f2f9c5c037b849ef3f127d052"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.152709 9476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:44:16.153358 9491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.153532 9476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:16.153644 9476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "eebe69e2d97a44deadeb9e43db4d5bdd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "bd0150a3436fca8e8f70ef80d8c6c371"
server_key_iv: "4753a90f2f9c5c037b849ef3f127d052"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.153723 9476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.161885 9476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.163849 9476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.164022 9476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.164264 9476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:16.164573 9476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:16.164637 9476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.164680 9476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:16.164707 9476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.170578 9476 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:37055
I20260924 17:44:16.170629 9604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:37055 every 8 connection(s)
I20260924 17:44:16.170953 9476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:44:16.178686 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9476
I20260924 17:44:16.178812 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:44:16.179086 1058 external_mini_cluster.cc:1428] Setting key 972b7a896945e0a4a55ac5aaf2ece95b
I20260924 17:44:16.181159 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.171772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:48283 (local address 127.1.8.190:39247)
0924 17:44:16.171836 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.171838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.172131 (+ 293us) server_negotiation.cc:407] Connection header received
0924 17:44:16.172692 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.172693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.172705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.172753 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.173724 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.174239 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.175306 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.175525 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.175607 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.177781 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:16.177783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:16.177783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:16.177791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:16.179667 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.180179 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.180180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.180180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.180240 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.180502 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.180503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.180503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.180627 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:16.180705 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:16.180980 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 17:44:16.181067 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:44:16.181519 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.171746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:48283)
0924 17:44:16.171955 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.172005 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.172554 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.172830 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.172834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.173411 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.173621 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.173626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.174344 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.174345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.175216 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.175218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.175628 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.175629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.175751 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.176324 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.176332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.177685 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:16.179777 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:16.179784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:16.179787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:16.180118 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:16.180313 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:16.180314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:16.180314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:16.180431 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:16.180696 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:16.180702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:16.180845 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 17:44:16.180962 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:44:16.181130 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
I20260924 17:44:16.181885 9605 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39247
I20260924 17:44:16.181998 9605 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:16.182180 9605 heartbeater.cc:511] Master 127.1.8.190:39247 requested a full tablet report, sending...
I20260924 17:44:16.182618 9280 ts_manager.cc:194] Registered new tserver with Master: eebe69e2d97a44deadeb9e43db4d5bdd (127.1.8.130:37055)
I20260924 17:44:16.183143 9280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:48283
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:44:16.205883 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39247
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:16.290297 9610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:16.290617 9610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:16.290668 9610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:16.291882 9610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:16.291952 9610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:16.292001 9610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:44:16.293495 9610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:16.293797 9610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:16.294040 9610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:16.299479 9616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.299479 9618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.299474 9615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.299952 9610 server_base.cc:1034] running on GCE node
I20260924 17:44:16.300258 9610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:16.300526 9610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:16.301754 9610 hybrid_clock.cc:648] HybridClock initialized: now 1790271856301734 us; error 39 us; skew 500 ppm
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:16.304683 9610 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:44:16.305287 9610 webserver.cc:460] Webserver started at http://127.1.8.131:36271/ using document root <none> and password file <none>
I20260924 17:44:16.305542 9610 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:16.305620 9610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:16.305789 9610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:16.307022 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "339ff06dae5c40f09e96d6e786e5b1dd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "261b33b3f21752a74b30676268b2cd98"
server_key_iv: "88d265bc07a745d181addaed928cd3a2"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.307453 9610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "339ff06dae5c40f09e96d6e786e5b1dd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "261b33b3f21752a74b30676268b2cd98"
server_key_iv: "88d265bc07a745d181addaed928cd3a2"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.308982 9610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:44:16.309995 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.310251 9610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:16.310365 9610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "339ff06dae5c40f09e96d6e786e5b1dd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "261b33b3f21752a74b30676268b2cd98"
server_key_iv: "88d265bc07a745d181addaed928cd3a2"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.310498 9610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.327765 9610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.329902 9610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.330076 9610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.330328 9610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:16.330674 9610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:16.330741 9610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.330780 9610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:16.330809 9610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.336865 9610 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:41831
I20260924 17:44:16.336942 9738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:41831 every 8 connection(s)
I20260924 17:44:16.337275 9610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:44:16.341001 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9610
I20260924 17:44:16.341099 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:44:16.341298 1058 external_mini_cluster.cc:1428] Setting key 0c311999d83d788d611a4d484298e7b2
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:44:16.347093 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.338072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:42893 (local address 127.1.8.190:39247)
0924 17:44:16.338145 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.338147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.338368 (+ 221us) server_negotiation.cc:407] Connection header received
0924 17:44:16.338938 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.338939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.338949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.339013 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.340075 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.340577 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.342025 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.342304 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.342373 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.344640 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:16.344642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:16.344642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:16.344649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:16.345842 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.346213 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.346214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.346214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.346262 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.346545 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.346546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.346546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.346666 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:16.346715 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:16.346965 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 17:44:16.347027 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:44:16.347306 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.338040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:42893)
0924 17:44:16.338236 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.338279 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.338844 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.339082 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.339086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.339746 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.339961 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.339967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.340716 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.340717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.341951 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.341954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.342443 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.342444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.342568 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.343159 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.343168 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.344551 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:16.345917 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:16.345923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:16.345925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:16.346145 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:16.346319 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:16.346320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:16.346320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:16.346493 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:16.346720 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:16.346724 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:16.346844 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 17:44:16.346924 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:44:16.347052 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":123,"threads_started":1}
I20260924 17:44:16.347834 9739 heartbeater.cc:347] Connected to a master server at 127.1.8.190:39247
I20260924 17:44:16.347944 9739 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:16.348146 9739 heartbeater.cc:511] Master 127.1.8.190:39247 requested a full tablet report, sending...
I20260924 17:44:16.348517 9280 ts_manager.cc:194] Registered new tserver with Master: 339ff06dae5c40f09e96d6e786e5b1dd (127.1.8.131:41831)
I20260924 17:44:16.348863 9280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:42893
I20260924 17:44:16.353185 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab.
Entry for principal oryx/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab.
WARNING: no policy specified for oryx/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab.
Entry for principal oryx/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab.
WARNING: no policy specified for oryx/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab.
Entry for principal oryx/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab.
I20260924 17:44:16.430323 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9342
I20260924 17:44:16.435243 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9476
I20260924 17:44:16.439760 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9610
I20260924 17:44:16.444314 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9248
I20260924 17:44:16.449580 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39247
--webserver_interface=127.1.8.190
--webserver_port=42611
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:16.532510 9748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:16.532784 9748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:16.532821 9748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:16.534016 9748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:16.534133 9748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:16.534160 9748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:16.534190 9748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:16.534222 9748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:16.535801 9748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=42611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:16.536096 9748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:16.536337 9748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:16.541350 9756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.541389 9754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.541389 9753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.541729 9748 server_base.cc:1034] running on GCE node
I20260924 17:44:16.542086 9748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:16.542460 9748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:16.543669 9748 hybrid_clock.cc:648] HybridClock initialized: now 1790271856543656 us; error 46 us; skew 500 ppm
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:16.546368 9748 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:44:16.546878 9748 webserver.cc:460] Webserver started at http://127.1.8.190:42611/ using document root <none> and password file <none>
I20260924 17:44:16.547142 9748 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:16.547226 9748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:16.548592 9748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:16.549294 9763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.549460 9748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:44:16.549572 9748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "dffe8d883a1f4425b43622fd522bbd37"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "6cfdf9648926e948feadca4a0ff76ae9"
server_key_iv: "47a493c144ed88dabeb6c7b40218bf45"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.549942 9748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.565753 9748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.568246 9748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.568421 9748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.572530 9748 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39247
I20260924 17:44:16.572570 9815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39247 every 8 connection(s)
I20260924 17:44:16.572978 9748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:44:16.574488 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9748
I20260924 17:44:16.574537 9816 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:44:16.574800 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:42857
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:39247
--webserver_port=46747
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:44:16.575317 9816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Bootstrap starting.
I20260924 17:44:16.586100 9816 log.cc:824] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:16.588364 9816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:44:16.588696 9816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Bootstrap complete.
I20260924 17:44:16.591877 9816 raft_consensus.cc:348] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:16.592185 9816 raft_consensus.cc:729] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffe8d883a1f4425b43622fd522bbd37, State: Initialized, Role: FOLLOWER
I20260924 17:44:16.592341 9816 consensus_queue.cc:260] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:16.592403 9816 raft_consensus.cc:388] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:44:16.592440 9816 raft_consensus.cc:482] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:44:16.592487 9816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:44:16.593767 9816 raft_consensus.cc:504] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:16.593875 9816 leader_election.cc:302] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dffe8d883a1f4425b43622fd522bbd37; no voters:
I20260924 17:44:16.594041 9816 leader_election.cc:288] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:44:16.594149 9820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:44:16.594300 9816 sys_catalog.cc:564] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:44:16.594364 9820 raft_consensus.cc:686] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [term 2 LEADER]: Becoming Leader. State: Replica: dffe8d883a1f4425b43622fd522bbd37, State: Running, Role: LEADER
I20260924 17:44:16.594532 9820 consensus_queue.cc:237] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } }
I20260924 17:44:16.594874 9821 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dffe8d883a1f4425b43622fd522bbd37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } } }
I20260924 17:44:16.594954 9821 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:16.595275 9822 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffe8d883a1f4425b43622fd522bbd37. Latest consensus state: current_term: 2 leader_uuid: "dffe8d883a1f4425b43622fd522bbd37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffe8d883a1f4425b43622fd522bbd37" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39247 } } }
I20260924 17:44:16.595342 9822 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:16.595552 9830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:44:16.596315 9830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:44:16.596439 9830 catalog_manager.cc:1259] Loaded cluster ID: 7794f67175374f90a1cec6f66ab01325
I20260924 17:44:16.596477 9830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:44:16.597597 9830 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:44:16.597777 9830 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dffe8d883a1f4425b43622fd522bbd37: Loaded TSK: 0
I20260924 17:44:16.597990 9830 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:44:16.663451 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:16.663745 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:16.663789 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:16.665024 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:16.665136 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:16.665189 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:44:16.666756 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.129.keytab
--principal=oryx/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:42857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=46747
--tserver_master_addrs=127.1.8.190:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:16.667047 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:16.667301 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:16.672447 9843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.672474 9818 server_base.cc:1034] running on GCE node
W20260924 17:44:16.672595 9844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.672451 9846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.672879 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:16.673166 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:16.674346 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790271856674335 us; error 35 us; skew 500 ppm
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:16.677053 9818 init.cc:377] Logged in from keytab as oryx/127.1.8.129@KRBTEST.COM (short username oryx)
I20260924 17:44:16.677534 9818 webserver.cc:460] Webserver started at http://127.1.8.129:46747/ using document root <none> and password file <none>
I20260924 17:44:16.677785 9818 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:16.677881 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:16.679435 9818 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:44:16.680200 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.680398 9818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:44:16.680501 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "fca575bd5e0e49b293be963222b4c64c"
format_stamp: "Formatted at 2026-09-24 17:44:15 on dist-test-slave-r7hf"
server_key: "e8114423c3de0848aed29a68ca68962c"
server_key_iv: "cafae6672fb44655b1d6b444d29be597"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.680886 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.704386 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.706871 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.707046 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.707295 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:16.707660 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:16.707715 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.707759 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:16.707785 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.714216 9818 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42857
I20260924 17:44:16.714264 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42857 every 8 connection(s)
I20260924 17:44:16.714735 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:44:16.720813 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9818
I20260924 17:44:16.721185 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:37055
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:39247
--webserver_port=43191
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.724601 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.715571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:50193)
0924 17:44:16.715753 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.715797 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.716283 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.716767 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.716770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.717395 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.717591 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.717596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.719371 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.719372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.720187 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.720190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.721045 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.721046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.721196 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.721808 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.721818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.724364 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
W20260924 17:44:16.724640 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.715652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:50193 (local address 127.1.8.190:39247)
0924 17:44:16.715879 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.715890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.715936 (+ 46us) server_negotiation.cc:407] Connection header received
0924 17:44:16.716503 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.716510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.716570 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.716730 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.717740 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.719265 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.720277 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.720862 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.720965 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.724492 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:50193: BlockingRecv error: recv got EOF from 127.1.8.129:50193 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
W20260924 17:44:16.725159 9967 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:39247 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.8.190:39247: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.729712 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.725398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:49459)
0924 17:44:16.725443 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.725456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.725497 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.725635 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.725636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.725763 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.725809 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.725810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.726492 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.726493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.727001 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.727002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.727385 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.727386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.727458 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.727927 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.727940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.729622 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 17:44:16.729768 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.725496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49459 (local address 127.1.8.190:39247)
0924 17:44:16.725533 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.725535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.725543 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:44:16.725564 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.725565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.725576 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.725626 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.725887 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.726382 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.727096 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.727299 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.727384 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.729703 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:49459: BlockingRecv error: recv got EOF from 127.1.8.129:49459 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.735492 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.729988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:36977)
0924 17:44:16.730141 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.730155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.730236 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.730374 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.730375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.730597 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.730688 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.730690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.731312 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.731313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.732353 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.732357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.732743 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.732745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.732821 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.733291 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.733296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.735392 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260924 17:44:16.735618 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.730029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:36977 (local address 127.1.8.190:39247)
0924 17:44:16.730068 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.730070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.730230 (+ 160us) server_negotiation.cc:407] Connection header received
0924 17:44:16.730300 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.730300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.730310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.730356 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.730769 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.731222 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.732450 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.732642 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.732696 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.735521 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:36977: BlockingRecv error: recv got EOF from 127.1.8.129:36977 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:44:16.735773 9967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:44:16.806705 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:16.806928 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:16.806954 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:16.808204 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:16.808252 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:16.808280 9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:44:16.809777 9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.130.keytab
--principal=oryx/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:37055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=43191
--tserver_master_addrs=127.1.8.190:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:16.810042 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:16.810297 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:16.815495 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.815495 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.815531 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.815820 9971 server_base.cc:1034] running on GCE node
I20260924 17:44:16.816144 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:16.816380 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:16.817534 9971 hybrid_clock.cc:648] HybridClock initialized: now 1790271856817502 us; error 48 us; skew 500 ppm
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:16.820381 9971 init.cc:377] Logged in from keytab as oryx/127.1.8.130@KRBTEST.COM (short username oryx)
I20260924 17:44:16.820892 9971 webserver.cc:460] Webserver started at http://127.1.8.130:43191/ using document root <none> and password file <none>
I20260924 17:44:16.821167 9971 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:16.821250 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:16.822891 9971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:44:16.823592 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.823838 9971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:16.823954 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "eebe69e2d97a44deadeb9e43db4d5bdd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "bd0150a3436fca8e8f70ef80d8c6c371"
server_key_iv: "4753a90f2f9c5c037b849ef3f127d052"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.824306 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.835840 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.837913 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.838091 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.838357 9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:16.838696 9971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:16.838728 9971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.838752 9971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:16.838804 9971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.845075 9971 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:37055
I20260924 17:44:16.845176 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:37055 every 8 connection(s)
I20260924 17:44:16.845517 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:44:16.847500 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 9971
I20260924 17:44:16.847805 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:41831
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:39247
--webserver_port=36271
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.854589 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.846269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:39999)
0924 17:44:16.846458 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.846503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.846996 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.847285 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.847289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.847964 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.848242 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.848248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.849291 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.849292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.850623 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.850626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.851135 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.851136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.851241 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.851882 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.851891 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.854356 (+ 2465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
W20260924 17:44:16.854598 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.846326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:39999 (local address 127.1.8.190:39247)
0924 17:44:16.846373 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.846375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.846626 (+ 251us) server_negotiation.cc:407] Connection header received
0924 17:44:16.847120 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.847121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.847136 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.847205 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.848351 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.849141 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.850712 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.851049 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.851131 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.854513 (+ 3382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:39999: BlockingRecv error: recv got EOF from 127.1.8.130:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:44:16.854902 10101 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:39247 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.8.190:39247: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.859555 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.855161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:33395)
0924 17:44:16.855214 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.855225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.855268 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.855392 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.855393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.855527 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.855568 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.855570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.856220 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.856221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.856736 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.856737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.857092 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.857092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.857152 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.857671 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.857675 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.859461 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 17:44:16.859609 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.855208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33395 (local address 127.1.8.190:39247)
0924 17:44:16.855257 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.855258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.855267 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:44:16.855333 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.855333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.855343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.855390 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.855669 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.856140 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.856831 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.857013 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.857092 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.859545 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:33395: BlockingRecv error: recv got EOF from 127.1.8.130:33395 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.864687 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.860085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:59137)
0924 17:44:16.860193 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.860204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.860259 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.860373 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.860374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.860525 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.860563 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.860565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.861158 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.861159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.861977 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.861979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.862292 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.862293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.862348 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.862797 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.862802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.864607 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 17:44:16.864779 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.860147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:59137 (local address 127.1.8.190:39247)
0924 17:44:16.860196 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.860197 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.860252 (+ 55us) server_negotiation.cc:407] Connection header received
0924 17:44:16.860317 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.860318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.860326 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.860369 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.860637 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.861086 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.862049 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.862223 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.862274 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.864697 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:59137: BlockingRecv error: recv got EOF from 127.1.8.130:59137 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 17:44:16.864818 10101 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:44:16.935451 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:16.935657 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:16.935681 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:16.936810 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:16.936846 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:16.936872 10104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:44:16.938252 10104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/krb5kdc/oryx_127.1.8.131.keytab
--principal=oryx/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:41831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=36271
--tserver_master_addrs=127.1.8.190:39247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:16.938519 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:16.938721 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:16.943903 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:16.943897 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.943930 10104 server_base.cc:1034] running on GCE node
W20260924 17:44:16.943897 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:16.944325 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:16.944561 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:16.945699 10104 hybrid_clock.cc:648] HybridClock initialized: now 1790271856945685 us; error 28 us; skew 500 ppm
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:16.948177 10104 init.cc:377] Logged in from keytab as oryx/127.1.8.131@KRBTEST.COM (short username oryx)
I20260924 17:44:16.948686 10104 webserver.cc:460] Webserver started at http://127.1.8.131:36271/ using document root <none> and password file <none>
I20260924 17:44:16.948933 10104 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:16.949016 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:16.950364 10104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:16.951012 10119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.951187 10104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.951290 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "339ff06dae5c40f09e96d6e786e5b1dd"
format_stamp: "Formatted at 2026-09-24 17:44:16 on dist-test-slave-r7hf"
server_key: "261b33b3f21752a74b30676268b2cd98"
server_key_iv: "88d265bc07a745d181addaed928cd3a2"
server_key_version: "encryptionkey@0"
I20260924 17:44:16.951671 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:16.965169 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:16.967303 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:16.967473 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:16.967721 10104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:16.968050 10104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:16.968106 10104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.968151 10104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:16.968179 10104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:16.974063 10104 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:41831
I20260924 17:44:16.974104 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:41831 every 8 connection(s)
I20260924 17:44:16.974534 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 17:44:16.981696 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.975325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:46065 (local address 127.1.8.190:39247)
0924 17:44:16.975399 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.975401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.975552 (+ 151us) server_negotiation.cc:407] Connection header received
0924 17:44:16.976058 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.976059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.976068 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.976135 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.977039 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.977531 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.978591 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.978782 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.978837 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.981650 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:46065: BlockingRecv error: recv got EOF from 127.1.8.131:46065 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:44:16.981706 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.975273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:46065)
0924 17:44:16.975431 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.975474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.975965 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.976195 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.976198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.976758 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.976957 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.976961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.977619 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.977620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.978502 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.978505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.978853 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.978853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.978959 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.979442 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.979450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.981562 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260924 17:44:16.981945 10233 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:39247 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.8.190:39247: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
I20260924 17:44:16.983819 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10104
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.987129 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.982178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:41091)
0924 17:44:16.982229 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.982241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.982287 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.982547 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.982548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.982660 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.982703 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.982704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.983452 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.983452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.984042 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.984045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.984629 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.984630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.984698 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.985204 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.985208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.987035 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:44:16.987244 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.982377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:41091 (local address 127.1.8.190:39247)
0924 17:44:16.982448 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.982450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.982457 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:44:16.982476 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.982477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.982484 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.982529 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.982800 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.983387 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.984228 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.984547 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.984613 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.987143 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:41091: BlockingRecv error: recv got EOF from 127.1.8.131:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:16 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:16.991879 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.987519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:38659)
0924 17:44:16.987575 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:16.987596 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:16.987632 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:16.987755 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:16.987755 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:16.987858 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:16.987883 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.987884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.988489 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.988490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.989095 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:16.989097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.989575 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.989576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:16.989651 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.989952 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:16.989956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:16.991827 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:44:16.991978 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:16.987580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:38659 (local address 127.1.8.190:39247)
0924 17:44:16.987644 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.987646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.987667 (+ 21us) server_negotiation.cc:407] Connection header received
0924 17:44:16.987687 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.987687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.987695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.987752 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.987965 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.988413 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.989207 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.989491 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.989556 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.991921 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:38659: BlockingRecv error: recv got EOF from 127.1.8.131:38659 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 17:44:16.992043 10233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:44:16.992568 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:16.984216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.1.8.190:39247)
0924 17:44:16.984358 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:16.984360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:16.984497 (+ 137us) server_negotiation.cc:407] Connection header received
0924 17:44:16.984546 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:16.984546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:16.984556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:16.984647 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:16.985394 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.986013 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.987111 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:16.987454 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:16.987521 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:16.988896 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:16.988901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:16.988901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:16.988916 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:16.990987 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.991370 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.991374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.991375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.991506 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:16.991746 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:16.991747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:16.991747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:16.992135 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:16.992299 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:16.992416 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 17:44:16.992507 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
Sep 24 17:44:17 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:17 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:17.741200 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:17.736265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:46207)
0924 17:44:17.736421 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:17.736434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:17.736507 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:17.736660 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:17.736661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:17.736820 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:17.736869 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:17.736870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.737555 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.737555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:17.738106 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:17.738108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.738482 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.738482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:17.738539 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:17.739004 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:17.739009 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:17.741070 (+ 2061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
W20260924 17:44:17.741276 10239 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:17.736375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:46207 (local address 127.1.8.190:39247)
0924 17:44:17.736528 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:17.736530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:17.736542 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:44:17.736566 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:17.736567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:17.736577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:17.736638 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:17.736941 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.737479 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.738186 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.738377 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.738479 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:17.741198 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:46207: BlockingRecv error: recv got EOF from 127.1.8.129:46207 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
Sep 24 17:44:17 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:17 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:17.873711 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:17.865235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:47775)
0924 17:44:17.865373 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:17.865395 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:17.865451 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:17.865643 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:17.865644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:17.865815 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:17.865864 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:17.865865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.866660 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.866662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:17.867216 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:17.867218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.867553 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.867554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:17.867594 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:17.871785 (+ 4191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:17.871793 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:17.873639 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260924 17:44:17.873831 10239 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:17.865290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:47775 (local address 127.1.8.190:39247)
0924 17:44:17.865373 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:17.865374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:17.865438 (+ 64us) server_negotiation.cc:407] Connection header received
0924 17:44:17.865506 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:17.865507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:17.865517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:17.865557 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:17.866007 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.866584 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.867299 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.867486 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.867540 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:17.873735 (+ 6195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:47775: BlockingRecv error: recv got EOF from 127.1.8.130:47775 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 17:44:17 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:17 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:17.997479 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:17.992480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:51153)
0924 17:44:17.992658 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:17.992668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:17.992726 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:17.992899 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:17.992900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:17.993079 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:17.993129 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:17.993131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.993820 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.993822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:17.994543 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:17.994546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.994932 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.994934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:17.994990 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:17.995564 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:17.995569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:17.997367 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
W20260924 17:44:17.997529 10239 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:17.992520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:51153 (local address 127.1.8.190:39247)
0924 17:44:17.992567 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:17.992569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:17.992713 (+ 144us) server_negotiation.cc:407] Connection header received
0924 17:44:17.992801 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:17.992802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:17.992815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:17.992892 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:17.993216 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.993721 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.994645 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:17.994844 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:17.994922 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:17.997461 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:51153: BlockingRecv error: recv got EOF from 127.1.8.131:51153 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 17:44:18 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:18 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:18.746850 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:18.741701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:34761)
0924 17:44:18.741866 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:18.741878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:18.742008 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:18.742218 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:18.742219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:18.742387 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:18.742514 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:18.742516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.743199 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.743199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:18.743795 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:18.743797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.744150 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.744151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:18.744196 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:18.744703 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:18.744708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:18.746791 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
W20260924 17:44:18.746996 10243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:18.741829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:34761 (local address 127.1.8.190:39247)
0924 17:44:18.741956 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:18.741958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:18.741968 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:44:18.742096 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:18.742097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:18.742108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:18.742170 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:18.742606 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.743124 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.743875 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.744081 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.744134 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:18.746896 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:34761: BlockingRecv error: recv got EOF from 127.1.8.129:34761 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
Sep 24 17:44:18 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:18 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:18.879179 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:18.874336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:59463)
0924 17:44:18.874472 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:18.874484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:18.874547 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:18.874685 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:18.874686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:18.874866 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:18.874911 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:18.874913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.875606 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.875607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:18.876121 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:18.876123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.876485 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.876486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:18.876544 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:18.877036 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:18.877041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:18.879073 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
W20260924 17:44:18.879216 10243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:18.874490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:59463 (local address 127.1.8.190:39247)
0924 17:44:18.874549 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:18.874551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:18.874563 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:44:18.874614 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:18.874615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:18.874628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:18.874683 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:18.874996 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.875499 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.876211 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.876401 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.876476 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:18.879154 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:59463: BlockingRecv error: recv got EOF from 127.1.8.130:59463 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 17:44:18 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:18 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:19.002770 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:18.997965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:39997)
0924 17:44:18.998121 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:18.998135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:18.998203 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:18.998332 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:18.998333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:18.998514 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:18.998562 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:18.998565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.999267 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.999268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:18.999812 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:18.999814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.000142 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.000143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:19.000190 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:19.000679 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:19.000685 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:19.002706 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
W20260924 17:44:19.002934 10243 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:18.998096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39997 (local address 127.1.8.190:39247)
0924 17:44:18.998143 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:18.998145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:18.998192 (+ 47us) server_negotiation.cc:407] Connection header received
0924 17:44:18.998257 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:18.998258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:18.998269 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:18.998308 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:18.998645 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:18.999134 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:18.999890 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.000069 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.000117 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:19.002845 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:39997: BlockingRecv error: recv got EOF from 127.1.8.131:39997 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 17:44:19 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:19 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:19.752158 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:19.747368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:47477)
0924 17:44:19.747536 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:19.747547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:19.747613 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:19.747752 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:19.747753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:19.747908 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:19.747957 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:19.747959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.748640 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.748640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:19.749156 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:19.749158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.749495 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.749496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:19.749534 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:19.750031 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:19.750036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:19.752032 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
W20260924 17:44:19.752278 10247 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:19.747488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:47477 (local address 127.1.8.190:39247)
0924 17:44:19.747601 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:19.747602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:19.747612 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:44:19.747673 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:19.747673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:19.747684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:19.747733 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:19.748030 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.748531 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.749237 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.749428 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.749481 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:19.752196 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:47477: BlockingRecv error: recv got EOF from 127.1.8.129:47477 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
Sep 24 17:44:19 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:19 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:19.884631 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:19.879721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:59291)
0924 17:44:19.879897 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:19.879908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:19.879970 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:19.880138 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:19.880138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:19.880300 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:19.880350 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:19.880352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.880986 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.880987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:19.881535 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:19.881537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.881907 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.881908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:19.881957 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:19.882429 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:19.882435 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:19.884561 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
W20260924 17:44:19.884706 10247 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:19.879833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:59291 (local address 127.1.8.190:39247)
0924 17:44:19.879900 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:19.879901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:19.879970 (+ 69us) server_negotiation.cc:407] Connection header received
0924 17:44:19.880017 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:19.880018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:19.880028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:19.880115 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:19.880416 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.880906 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.881641 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:19.881818 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:19.881905 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:19.884659 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:59291: BlockingRecv error: recv got EOF from 127.1.8.130:59291 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 17:44:20 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:20 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:20.008328 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:20.003456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:48967)
0924 17:44:20.003690 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:20.003702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:20.003787 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:20.003951 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:20.003952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:20.004227 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:20.004284 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:20.004286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.005150 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.005151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:20.005693 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:20.005695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.006018 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.006019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:20.006057 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:20.006532 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:20.006537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:20.008261 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":156,"threads_started":1}
W20260924 17:44:20.008513 10247 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:20.003585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:48967 (local address 127.1.8.190:39247)
0924 17:44:20.003681 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:20.003684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:20.003781 (+ 97us) server_negotiation.cc:407] Connection header received
0924 17:44:20.003850 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:20.003851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:20.003872 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:20.003944 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:20.004379 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.005071 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.005771 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.005950 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.006004 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:20.008436 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:48967: BlockingRecv error: recv got EOF from 127.1.8.131:48967 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 17:44:20 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:20 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:20.757292 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:20.752617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:50655)
0924 17:44:20.752767 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:20.752777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:20.752821 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:20.752986 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:20.752987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:20.753144 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:20.753193 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:20.753194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.753949 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.753951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:20.754555 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:20.754557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.754959 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.754960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:20.755016 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:20.755466 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:20.755470 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:20.757223 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
W20260924 17:44:20.757432 10251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:20.752668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:50655 (local address 127.1.8.190:39247)
0924 17:44:20.752797 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:20.752799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:20.752828 (+ 29us) server_negotiation.cc:407] Connection header received
0924 17:44:20.752888 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:20.752889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:20.752904 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:20.752970 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:20.753256 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.753856 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.754656 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.754882 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.754960 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:20.757321 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:50655: BlockingRecv error: recv got EOF from 127.1.8.129:50655 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
Sep 24 17:44:20 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:20 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:20.889794 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:20.885201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:45141)
0924 17:44:20.885336 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:20.885347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:20.885408 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:20.885533 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:20.885533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:20.885699 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:20.885784 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:20.885786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.886481 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.886483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:20.887006 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:20.887008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.887353 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.887354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:20.887393 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:20.887918 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:20.887924 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:20.889735 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
W20260924 17:44:20.889947 10251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:20.885254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:45141 (local address 127.1.8.190:39247)
0924 17:44:20.885321 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:20.885323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:20.885395 (+ 72us) server_negotiation.cc:407] Connection header received
0924 17:44:20.885462 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:20.885463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:20.885473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:20.885517 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:20.885879 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.886384 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.887103 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:20.887287 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:20.887340 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:20.889864 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:45141: BlockingRecv error: recv got EOF from 127.1.8.130:45141 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 17:44:21 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:21 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:21.014329 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:21.009080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:56389)
0924 17:44:21.009297 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:21.009312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:21.009418 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:21.009586 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:21.009587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:21.009765 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:21.009810 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:21.009812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.010561 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.010562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:21.011104 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:21.011106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.011488 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.011489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:21.011541 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:21.012043 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:21.012048 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:21.014238 (+ 2190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
W20260924 17:44:21.014529 10251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:21.009252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:56389 (local address 127.1.8.190:39247)
0924 17:44:21.009323 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:21.009325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:21.009399 (+ 74us) server_negotiation.cc:407] Connection header received
0924 17:44:21.009477 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:21.009479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:21.009493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:21.009571 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:21.009898 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.010479 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.011193 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.011386 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.011488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:21.014368 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:56389: BlockingRecv error: recv got EOF from 127.1.8.131:56389 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 17:44:21 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:21 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:21.762945 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:21.757799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:39415)
0924 17:44:21.757956 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:21.757966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:21.758035 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:21.758177 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:21.758178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:21.758340 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:21.758451 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:21.758454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.759425 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.759426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:21.760091 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:21.760094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.760456 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.760457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:21.760506 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:21.760953 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:21.760957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:21.762875 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
W20260924 17:44:21.763044 10255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:21.757887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:39415 (local address 127.1.8.190:39247)
0924 17:44:21.758033 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:21.758035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:21.758045 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:44:21.758090 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:21.758090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:21.758100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:21.758158 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:21.758530 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.759320 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.760181 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.760372 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.760430 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:21.762968 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:39415: BlockingRecv error: recv got EOF from 127.1.8.129:39415 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
Sep 24 17:44:21 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:21 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:21.895226 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:21.890314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:47313)
0924 17:44:21.890477 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:21.890489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:21.890552 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:21.890730 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:21.890731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:21.890916 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:21.890972 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:21.890974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.891687 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.891689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:21.892214 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:21.892215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.892595 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.892596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:21.892659 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:21.893096 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:21.893101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:21.895127 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
W20260924 17:44:21.895278 10255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:21.890476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:47313 (local address 127.1.8.190:39247)
0924 17:44:21.890530 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:21.890532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:21.890542 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:44:21.890606 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:21.890607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:21.890616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:21.890665 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:21.891097 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.891597 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.892304 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:21.892500 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:21.892593 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:21.895214 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:47313: BlockingRecv error: recv got EOF from 127.1.8.130:47313 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 17:44:22 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:22 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:22.019712 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:22.014985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:37087)
0924 17:44:22.015196 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:22.015208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:22.015279 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:22.015475 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:22.015477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:22.015648 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:22.015697 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:22.015699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.016367 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.016368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:22.016906 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:22.016908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.017229 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.017230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:22.017270 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:22.017726 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:22.017731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:22.019650 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":128,"threads_started":1}
W20260924 17:44:22.019829 10255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:22.015055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:37087 (local address 127.1.8.190:39247)
0924 17:44:22.015127 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:22.015129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:22.015280 (+ 151us) server_negotiation.cc:407] Connection header received
0924 17:44:22.015337 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:22.015338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:22.015352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:22.015392 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:22.015779 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.016284 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.016986 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.017165 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.017217 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:22.019750 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:37087: BlockingRecv error: recv got EOF from 127.1.8.131:37087 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 17:44:22 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:22 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:22.768088 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:22.763490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:56721)
0924 17:44:22.763689 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:22.763701 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:22.763761 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:22.763881 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:22.763882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:22.764045 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:22.764093 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:22.764095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.764740 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.764740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:22.765279 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:22.765281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.765612 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.765613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:22.765654 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:22.766094 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:22.766098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:22.767994 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":129,"threads_started":1}
W20260924 17:44:22.768165 10259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:22.763594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56721 (local address 127.1.8.190:39247)
0924 17:44:22.763707 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:22.763711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:22.763766 (+ 55us) server_negotiation.cc:407] Connection header received
0924 17:44:22.763815 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:22.763815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:22.763825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:22.763869 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:22.764157 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.764664 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.765355 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.765541 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.765596 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:22.768111 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:56721: BlockingRecv error: recv got EOF from 127.1.8.129:56721 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
Sep 24 17:44:22 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:22 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:22.906648 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:22.895791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:55871)
0924 17:44:22.895935 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:22.895948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:22.896033 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:22.896186 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:22.896187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:22.896349 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:22.896394 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:22.896395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.897035 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.897035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:22.897557 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:22.897559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.897915 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.897916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:22.897953 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:22.904567 (+ 6614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:22.904574 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:22.906585 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
W20260924 17:44:22.906795 10259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:22.895871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:55871 (local address 127.1.8.190:39247)
0924 17:44:22.895946 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:22.895949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:22.896005 (+ 56us) server_negotiation.cc:407] Connection header received
0924 17:44:22.896104 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:22.896104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:22.896115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:22.896164 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:22.896457 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.896947 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.897651 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:22.897844 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:22.897905 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:22.906713 (+ 8808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:55871: BlockingRecv error: recv got EOF from 127.1.8.130:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 17:44:23 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:23 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:23.025161 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:23.020286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:58625)
0924 17:44:23.020491 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:23.020509 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:23.020591 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:23.020711 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:23.020712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:23.020880 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:23.020928 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:23.020930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.021565 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.021565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:23.022150 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:23.022152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.022531 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.022532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:23.022579 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:23.023171 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:23.023176 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:23.025101 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
W20260924 17:44:23.025297 10259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:23.020420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:58625 (local address 127.1.8.190:39247)
0924 17:44:23.020493 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:23.020495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:23.020587 (+ 92us) server_negotiation.cc:407] Connection header received
0924 17:44:23.020645 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:23.020646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:23.020657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:23.020706 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:23.020985 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.021472 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.022228 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.022437 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.022521 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:23.025200 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:58625: BlockingRecv error: recv got EOF from 127.1.8.131:58625 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 17:44:23 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:23 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:23.773156 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:23.768679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:37517)
0924 17:44:23.768862 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:23.768872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:23.768928 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:23.769064 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:23.769065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:23.769222 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:23.769270 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:23.769272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.769933 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.769933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:23.770508 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:23.770509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.770841 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.770842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:23.770884 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:23.771364 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:23.771368 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:23.773097 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":120,"threads_started":1}
W20260924 17:44:23.773269 10263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:23.768803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:37517 (local address 127.1.8.190:39247)
0924 17:44:23.768911 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:23.768913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:23.768925 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:44:23.768986 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:23.768987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:23.768996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:23.769048 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:23.769343 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.769856 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.770588 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.770778 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.770831 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:23.773192 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:37517: BlockingRecv error: recv got EOF from 127.1.8.129:37517 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
Sep 24 17:44:23 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:23 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:23.911435 10264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:23.907138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:58379)
0924 17:44:23.907270 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:23.907280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:23.907337 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:23.907446 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:23.907446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:23.907601 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:23.907648 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:23.907650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.908256 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.908257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:23.908796 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:23.908797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.909109 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.909110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:23.909147 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:23.909585 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:23.909590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:23.911374 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260924 17:44:23.911583 10263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:23.907215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:58379 (local address 127.1.8.190:39247)
0924 17:44:23.907276 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:23.907278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:23.907344 (+ 66us) server_negotiation.cc:407] Connection header received
0924 17:44:23.907382 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:23.907383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:23.907393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:23.907442 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:23.907703 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.908182 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.908866 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:23.909043 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:23.909097 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:23.911484 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:58379: BlockingRecv error: recv got EOF from 127.1.8.130:58379 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 17:44:24 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:24 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:24.031032 10265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:24.025780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:58403)
0924 17:44:24.025918 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:24.025937 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:24.025988 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:24.026210 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:24.026211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:24.026386 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:24.026456 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:24.026458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.027339 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.027341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:24.027926 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:24.027928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.028321 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.028322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:24.028377 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:24.028867 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:24.028871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:24.030903 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260924 17:44:24.031044 10263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:24.025995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:58403 (local address 127.1.8.190:39247)
0924 17:44:24.026084 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:24.026090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:24.026098 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:44:24.026122 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:24.026122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:24.026134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:24.026203 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:24.026589 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.027255 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.028048 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.028246 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.028323 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:24.030988 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:58403: BlockingRecv error: recv got EOF from 127.1.8.131:58403 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 17:44:24 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:24 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:24.778139 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:24.773632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:45893)
0924 17:44:24.773866 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:24.773884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:24.773944 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:24.774096 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:24.774097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:24.774255 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:24.774302 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:24.774304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.774998 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.774998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:24.775524 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:24.775526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.775870 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.775870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:24.775909 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:24.776355 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:24.776359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:24.778082 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":186,"threads_started":1}
W20260924 17:44:24.778276 10267 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:24.773749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:45893 (local address 127.1.8.190:39247)
0924 17:44:24.773903 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:24.773904 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:24.773930 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:44:24.773998 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:24.773999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:24.774009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:24.774074 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:24.774368 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.774925 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.775603 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.775797 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.775857 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:24.778186 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:45893: BlockingRecv error: recv got EOF from 127.1.8.129:45893 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
Sep 24 17:44:24 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:24 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:24.916698 10268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:24.911918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:52999)
0924 17:44:24.912089 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:24.912110 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:24.912178 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:24.912314 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:24.912314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:24.912467 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:24.912521 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:24.912523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.913289 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.913290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:24.913813 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:24.913814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.914161 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.914162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:24.914200 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:24.914667 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:24.914671 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:24.916640 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
W20260924 17:44:24.916798 10267 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:24.912042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:52999 (local address 127.1.8.190:39247)
0924 17:44:24.912117 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:24.912119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:24.912162 (+ 43us) server_negotiation.cc:407] Connection header received
0924 17:44:24.912237 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:24.912238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:24.912248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:24.912289 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:24.912586 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.913216 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.913894 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:24.914091 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:24.914149 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:24.916730 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:52999: BlockingRecv error: recv got EOF from 127.1.8.130:52999 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 17:44:25 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:25 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:25.036527 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:25.031597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:55341)
0924 17:44:25.031782 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:25.031794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:25.031895 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:25.032074 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:25.032075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:25.032257 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:25.032299 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:25.032301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.033011 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.033012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:25.033598 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:25.033599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.033988 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.033989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:25.034040 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:25.034579 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:25.034584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:25.036460 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":89,"threads_started":1}
W20260924 17:44:25.036630 10267 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:25.031691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:55341 (local address 127.1.8.190:39247)
0924 17:44:25.031768 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:25.031771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:25.031874 (+ 103us) server_negotiation.cc:407] Connection header received
0924 17:44:25.031968 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:25.031969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:25.031994 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:25.032060 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:25.032390 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.032889 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.033698 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.033896 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.033997 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:25.036561 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:55341: BlockingRecv error: recv got EOF from 127.1.8.131:55341 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 17:44:25 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:25 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:25.783284 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:25.778815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:59949)
0924 17:44:25.778953 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:25.778965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:25.779031 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:25.779177 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:25.779178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:25.779339 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:25.779385 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:25.779387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.780027 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.780027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:25.780544 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:25.780546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.780862 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.780862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:25.780899 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:25.781332 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:25.781336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:25.783206 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
W20260924 17:44:25.783440 10271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:25.778873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:59949 (local address 127.1.8.190:39247)
0924 17:44:25.779003 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:25.779023 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:25.779034 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:44:25.779078 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:25.779078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:25.779088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:25.779156 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:25.779450 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.779953 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.780616 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.780798 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.780848 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:25.783314 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:59949: BlockingRecv error: recv got EOF from 127.1.8.129:59949 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
Sep 24 17:44:25 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:25 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:25.921902 10272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:25.917299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:58035)
0924 17:44:25.917444 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:25.917455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:25.917509 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:25.917639 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:25.917639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:25.917799 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:25.917844 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:25.917845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.918527 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.918528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:25.919082 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:25.919084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.919516 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.919518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:25.919566 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:25.920021 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:25.920026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:25.921842 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260924 17:44:25.921989 10271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:25.917388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:58035 (local address 127.1.8.190:39247)
0924 17:44:25.917488 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:25.917490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:25.917519 (+ 29us) server_negotiation.cc:407] Connection header received
0924 17:44:25.917563 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:25.917563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:25.917574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:25.917619 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:25.917913 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.918461 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.919170 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:25.919387 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:25.919447 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:25.921942 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:58035: BlockingRecv error: recv got EOF from 127.1.8.130:58035 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 17:44:26 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:26 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:26.041626 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:26.036968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:35977)
0924 17:44:26.037152 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:26.037163 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:26.037230 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:26.037392 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:26.037393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:26.037602 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:26.037653 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:26.037655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.038365 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.038367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:26.038919 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:26.038920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.039279 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.039280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:26.039337 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:26.039782 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:26.039786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:26.041530 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
W20260924 17:44:26.041683 10271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:26.037061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:35977 (local address 127.1.8.190:39247)
0924 17:44:26.037129 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:26.037131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:26.037211 (+ 80us) server_negotiation.cc:407] Connection header received
0924 17:44:26.037298 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:26.037301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:26.037318 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:26.037383 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:26.037740 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.038266 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.039022 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.039205 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.039282 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:26.041616 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:35977: BlockingRecv error: recv got EOF from 127.1.8.131:35977 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 17:44:26 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:26 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:26.788632 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:26.783935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:36683)
0924 17:44:26.784090 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:26.784101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:26.784163 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:26.784351 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:26.784351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:26.784516 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:26.784565 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:26.784567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.785266 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.785267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:26.785815 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:26.785817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.786188 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.786188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:26.786229 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:26.786724 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:26.786728 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:26.788538 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
W20260924 17:44:26.788702 10275 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:26.783990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:36683 (local address 127.1.8.190:39247)
0924 17:44:26.784121 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:26.784123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:26.784153 (+ 30us) server_negotiation.cc:407] Connection header received
0924 17:44:26.784255 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:26.784256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:26.784270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:26.784338 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:26.784637 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.785159 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.785906 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.786114 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.786170 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:26.788631 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:36683: BlockingRecv error: recv got EOF from 127.1.8.129:36683 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
Sep 24 17:44:26 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:26 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:26.927614 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:26.922615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:35983)
0924 17:44:26.922780 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:26.922792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:26.922861 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:26.923024 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:26.923025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:26.923192 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:26.923237 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:26.923239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.923887 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.923888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:26.924403 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:26.924405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.924722 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.924722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:26.924761 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:26.925261 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:26.925266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:26.927554 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
W20260924 17:44:26.927824 10275 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:26.922713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:35983 (local address 127.1.8.190:39247)
0924 17:44:26.922768 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:26.922771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:26.922864 (+ 93us) server_negotiation.cc:407] Connection header received
0924 17:44:26.922907 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:26.922908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:26.922918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:26.923000 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:26.923300 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.923812 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.924476 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:26.924657 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:26.924708 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:26.927671 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:35983: BlockingRecv error: recv got EOF from 127.1.8.130:35983 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 17:44:27 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:27 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:27.047056 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:27.042200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:38493)
0924 17:44:27.042359 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:27.042368 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:27.042487 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:27.042708 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:27.042709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:27.042881 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:27.042928 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:27.042930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.043597 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.043598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:27.044163 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:27.044165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.044483 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.044484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:27.044525 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:27.044999 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:27.045004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:27.046953 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260924 17:44:27.047092 10275 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:27.042323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:38493 (local address 127.1.8.190:39247)
0924 17:44:27.042423 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:27.042427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:27.042446 (+ 19us) server_negotiation.cc:407] Connection header received
0924 17:44:27.042563 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:27.042563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:27.042575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:27.042631 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:27.043001 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.043499 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.044240 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.044415 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.044467 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:27.047032 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:38493: BlockingRecv error: recv got EOF from 127.1.8.131:38493 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 17:44:27 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:27 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:27.800069 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:27.789218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:52445)
0924 17:44:27.789387 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:27.789399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:27.789460 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:27.789649 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:27.789651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:27.789833 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:27.789883 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:27.789884 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.790558 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.790559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:27.791085 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:27.791087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.791417 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.791418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:27.791473 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:27.798098 (+ 6625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:27.798104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:27.800007 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
W20260924 17:44:27.800203 10279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:27.789344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:52445 (local address 127.1.8.190:39247)
0924 17:44:27.789476 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:27.789478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:27.789511 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:44:27.789535 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:27.789536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:27.789546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:27.789601 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:27.789958 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.790485 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.791163 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.791350 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.791404 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:27.800113 (+ 8709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:52445: BlockingRecv error: recv got EOF from 127.1.8.129:52445 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
Sep 24 17:44:27 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:27 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:27.937554 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:27.928151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:35679)
0924 17:44:27.928300 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:27.928312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:27.928411 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:27.928572 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:27.928573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:27.928760 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:27.928807 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:27.928809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.929527 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.929528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:27.930073 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:27.930075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.930567 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.930569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:27.930621 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:27.935543 (+ 4922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:27.935548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:27.937495 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260924 17:44:27.937706 10279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:27.928189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:35679 (local address 127.1.8.190:39247)
0924 17:44:27.928258 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:27.928261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:27.928400 (+ 139us) server_negotiation.cc:407] Connection header received
0924 17:44:27.928480 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:27.928481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:27.928501 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:27.928564 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:27.928895 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.929413 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.930175 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:27.930379 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:27.930474 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:27.937630 (+ 7156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:35679: BlockingRecv error: recv got EOF from 127.1.8.130:35679 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 17:44:28 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:28 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:28.052733 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:28.047610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:56835)
0924 17:44:28.047747 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:28.047759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:28.047816 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:28.048114 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:28.048115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:28.048273 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:28.048320 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:28.048321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.049080 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.049081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:28.049596 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:28.049597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.050044 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.050045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:28.050095 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:28.050721 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:28.050726 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:28.052671 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
W20260924 17:44:28.053095 10279 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:28.047668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:56835 (local address 127.1.8.190:39247)
0924 17:44:28.047720 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:28.047723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:28.047808 (+ 85us) server_negotiation.cc:407] Connection header received
0924 17:44:28.047864 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:28.047865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:28.047880 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:28.048105 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:28.048404 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.049008 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.049681 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.049950 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.050015 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:28.052976 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:56835: BlockingRecv error: recv got EOF from 127.1.8.131:56835 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 17:44:28 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:28 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:28.805552 10282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:28.800700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:46185)
0924 17:44:28.800886 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:28.800898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:28.800963 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:28.801113 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:28.801114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:28.801276 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:28.801324 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:28.801326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.802117 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.802118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:28.802674 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:28.802676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.803052 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.803052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:28.803091 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:28.803655 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:28.803659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:28.805491 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
W20260924 17:44:28.805703 10283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:28.800829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:46185 (local address 127.1.8.190:39247)
0924 17:44:28.800973 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:28.800975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:28.800986 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:44:28.801012 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:28.801012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:28.801030 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:28.801091 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:28.801404 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.802001 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.802742 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.802982 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.803038 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:28.805613 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:46185: BlockingRecv error: recv got EOF from 127.1.8.129:46185 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
Sep 24 17:44:28 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:28 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:28.942517 10284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:28.938024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:34227)
0924 17:44:28.938154 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:28.938166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:28.938243 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:28.938382 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:28.938383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:28.938556 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:28.938604 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:28.938605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.939250 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.939251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:28.939797 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:28.939799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.940122 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.940123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:28.940161 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:28.940588 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:28.940592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:28.942404 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260924 17:44:28.942602 10283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:28.938073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:34227 (local address 127.1.8.190:39247)
0924 17:44:28.938169 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:28.938170 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:28.938224 (+ 54us) server_negotiation.cc:407] Connection header received
0924 17:44:28.938310 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:28.938311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:28.938329 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:28.938372 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:28.938681 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.939172 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.939876 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:28.940056 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:28.940107 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:28.942539 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:34227: BlockingRecv error: recv got EOF from 127.1.8.130:34227 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 17:44:29 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:29 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:29.058080 10285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:29.053411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:43871)
0924 17:44:29.053566 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:29.053579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:29.053657 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:29.053797 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:29.053798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:29.053963 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:29.054013 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:29.054014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.054772 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.054774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:29.055347 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:29.055350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.055765 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.055767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:29.055830 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:29.056287 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:29.056301 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:29.058002 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
W20260924 17:44:29.058198 10283 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:29.053546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:43871 (local address 127.1.8.190:39247)
0924 17:44:29.053614 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:29.053615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:29.053639 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:44:29.053713 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:29.053714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:29.053726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:29.053775 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:29.054074 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.054668 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.055426 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.055661 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.055715 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:29.058119 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:43871: BlockingRecv error: recv got EOF from 127.1.8.131:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 17:44:29 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:29 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:29.811136 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:29.806142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:56301)
0924 17:44:29.806306 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:29.806318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:29.806434 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:29.806651 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:29.806652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:29.806875 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:29.806946 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:29.806949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.807626 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.807627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:29.808146 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:29.808147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.808525 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.808526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:29.808570 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:29.809102 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:29.809107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:29.811077 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
W20260924 17:44:29.811272 10287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:29.806287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56301 (local address 127.1.8.190:39247)
0924 17:44:29.806434 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:29.806436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:29.806449 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:44:29.806528 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:29.806529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:29.806542 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:29.806629 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:29.807041 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.807550 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.808245 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.808448 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.808510 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:29.811184 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:56301: BlockingRecv error: recv got EOF from 127.1.8.129:56301 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":58,"threads_started":1}
Sep 24 17:44:29 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:29 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:29.947846 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:29.943107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:34001)
0924 17:44:29.943296 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:29.943311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:29.943383 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:29.943578 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:29.943579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:29.943744 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:29.943793 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:29.943794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.944500 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.944500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:29.945025 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:29.945027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.945374 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.945375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:29.945414 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:29.945937 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:29.945947 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:29.947774 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":80,"threads_started":1}
W20260924 17:44:29.947979 10287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:29.943280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:34001 (local address 127.1.8.190:39247)
0924 17:44:29.943355 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:29.943357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:29.943366 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:44:29.943490 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:29.943491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:29.943502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:29.943557 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:29.943862 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.944401 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.945105 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:29.945304 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:29.945360 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:29.947905 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:34001: BlockingRecv error: recv got EOF from 127.1.8.130:34001 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 17:44:30 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:30 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:30.063350 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:30.058763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:35663)
0924 17:44:30.058889 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:30.058900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:30.058983 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:30.059170 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:30.059171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:30.059394 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:30.059443 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:30.059444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.060174 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.060174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:30.060688 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:30.060689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.061037 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.061038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:30.061077 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:30.061530 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:30.061535 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:30.063279 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
W20260924 17:44:30.063508 10287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:30.058795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:35663 (local address 127.1.8.190:39247)
0924 17:44:30.058850 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:30.058852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:30.058983 (+ 131us) server_negotiation.cc:407] Connection header received
0924 17:44:30.059067 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:30.059068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:30.059083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:30.059148 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:30.059516 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.060096 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.060765 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.060968 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.061023 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:30.063370 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:35663: BlockingRecv error: recv got EOF from 127.1.8.131:35663 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 17:44:30 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:30 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:30.816582 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:30.811741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:56129)
0924 17:44:30.811986 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:30.811998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:30.812068 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:30.812197 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:30.812198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:30.812364 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:30.812412 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:30.812414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.813072 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.813073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:30.813617 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:30.813619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.813969 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.813970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:30.814009 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:30.814523 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:30.814528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:30.816511 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":194,"threads_started":1}
W20260924 17:44:30.816735 10290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:30.811813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56129 (local address 127.1.8.190:39247)
0924 17:44:30.811964 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:30.811966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:30.812072 (+ 106us) server_negotiation.cc:407] Connection header received
0924 17:44:30.812112 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:30.812113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:30.812123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:30.812192 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:30.812486 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.812986 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.813705 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.813897 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.813954 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:30.816645 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:56129: BlockingRecv error: recv got EOF from 127.1.8.129:56129 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
Sep 24 17:44:30 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:30 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:30.953068 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:30.948386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:39607)
0924 17:44:30.948572 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:30.948583 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:30.948637 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:30.948788 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:30.948788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:30.948946 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:30.948994 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:30.948995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.949617 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.949618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:30.950142 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:30.950144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.950470 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.950471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:30.950509 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:30.951023 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:30.951027 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:30.952970 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
W20260924 17:44:30.953110 10290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:30.948514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:39607 (local address 127.1.8.190:39247)
0924 17:44:30.948579 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:30.948581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:30.948626 (+ 45us) server_negotiation.cc:407] Connection header received
0924 17:44:30.948708 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:30.948709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:30.948719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:30.948762 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:30.949065 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.949545 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.950220 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:30.950421 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:30.950470 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:30.953047 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:39607: BlockingRecv error: recv got EOF from 127.1.8.130:39607 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 17:44:31 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:31 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.131@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:31.068241 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:31.063807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.131:35149)
0924 17:44:31.063982 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:31.063994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:31.064061 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:31.064189 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:31.064190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:31.064350 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:31.064399 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:31.064400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.065018 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.065018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:31.065523 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:31.065524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.065846 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.065846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:31.065884 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:31.066338 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:31.066342 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:31.068181 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
W20260924 17:44:31.068358 10290 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:31.063883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:35149 (local address 127.1.8.190:39247)
0924 17:44:31.063973 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:31.063976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:31.064045 (+ 69us) server_negotiation.cc:407] Connection header received
0924 17:44:31.064115 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:31.064116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:31.064127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:31.064165 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:31.064465 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.064949 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.065602 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.065781 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.065834 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:31.068279 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.131:35149: BlockingRecv error: recv got EOF from 127.1.8.131:35149 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 17:44:31 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:31 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.129@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:31.822894 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:31.817323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.129:50275)
0924 17:44:31.817467 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:31.817481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:31.817560 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:31.817759 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:31.817761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:31.817929 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:31.817981 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:31.817983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.818694 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.818695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:31.819316 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:31.819319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.819718 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.819720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:31.819786 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:31.820413 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:31.820419 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:31.822828 (+ 2409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260924 17:44:31.823216 10295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:31.817390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:50275 (local address 127.1.8.190:39247)
0924 17:44:31.817533 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:31.817535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:31.817546 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:44:31.817612 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:31.817613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:31.817623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:31.817686 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:31.818063 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.818596 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.819407 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.819619 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.819676 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:31.823152 (+ 3476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.129:50275: BlockingRecv error: recv got EOF from 127.1.8.129:50275 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
Sep 24 17:44:31 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 17:44:31 dist-test-slave-r7hf krb5kdc[9235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.8.130@KRBTEST.COM for oryx/127.1.8.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 17:44:31.958511 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:31.953687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39247 (local address 127.1.8.130:48131)
0924 17:44:31.953814 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:31.953826 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:31.953894 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:31.954051 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:31.954052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:31.954328 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:31.954456 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:31.954459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.955190 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.955191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:31.955850 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:31.955853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.956209 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.956209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:31.956249 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:31.956702 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:31.956706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:31.958359 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.8.190:39247: Server oryx/127.1.8.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260924 17:44:31.958561 10295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:31.953727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:48131 (local address 127.1.8.190:39247)
0924 17:44:31.953770 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:31.953772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:31.953940 (+ 168us) server_negotiation.cc:407] Connection header received
0924 17:44:31.953966 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:31.953967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:31.953977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:31.954027 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:31.954537 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.955052 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.955946 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:31.956137 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:31.956196 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:31.958513 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.8.130:48131: BlockingRecv error: recv got EOF from 127.1.8.130:48131 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:44:31.985749 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9818
I20260924 17:44:31.990477 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9971
I20260924 17:44:31.994974 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10104
I20260924 17:44:31.999388 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 9748
2026-09-24T17:44:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17486 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T17:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:32Z Disabled control of system clock
I20260924 17:44:32.027772 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41183
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:41183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 17:44:32.108827 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.109061 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.109081 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.110634 10300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:32.110697 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.110710 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.110724 10300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:32.110860 10300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:32.112529 10300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:41183
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.112829 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.113160 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.118358 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.118532 10300 server_base.cc:1034] running on GCE node
W20260924 17:44:32.118378 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.118371 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.118901 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.119227 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.120437 10300 hybrid_clock.cc:648] HybridClock initialized: now 1790271872120420 us; error 43 us; skew 500 ppm
I20260924 17:44:32.121912 10300 webserver.cc:460] Webserver started at http://127.1.8.190:42463/ using document root <none> and password file <none>
I20260924 17:44:32.122169 10300 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.122251 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.122460 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.123591 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "6e33f1c70f0c4db58890deed8f12df71"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "621cf17157c433c415ddd59e18406dba"
server_key_iv: "09357c4bbbbf196207f8ab1b74c7fdd3"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.123976 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "6e33f1c70f0c4db58890deed8f12df71"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "621cf17157c433c415ddd59e18406dba"
server_key_iv: "09357c4bbbbf196207f8ab1b74c7fdd3"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.125326 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:44:32.126072 10314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.126257 10300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:32.126358 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "6e33f1c70f0c4db58890deed8f12df71"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "621cf17157c433c415ddd59e18406dba"
server_key_iv: "09357c4bbbbf196207f8ab1b74c7fdd3"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.126451 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.146209 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.146646 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.146814 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.150676 10300 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:41183
I20260924 17:44:32.150712 10366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:41183 every 8 connection(s)
I20260924 17:44:32.151088 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:44:32.152055 10367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:32.152305 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10300
I20260924 17:44:32.152410 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:44:32.152633 1058 external_mini_cluster.cc:1428] Setting key 4836db5b7dee19ee3ff7ffb4326a4790
I20260924 17:44:32.155066 10367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71: Bootstrap starting.
I20260924 17:44:32.155825 10367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:32.156131 10367 log.cc:824] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:32.156826 10367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71: No bootstrap required, opened a new log
I20260924 17:44:32.158636 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.153225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.1.8.190:41183)
0924 17:44:32.153439 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.153446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.153471 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:44:32.153887 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.153893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.153955 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.154077 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.154306 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.155238 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.156307 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.156949 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.157064 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.157729 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.157779 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.157779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.157948 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.157993 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.158004 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.158062 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.158174 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.158285 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260924 17:44:32.158742 10367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e33f1c70f0c4db58890deed8f12df71" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41183 } }
I20260924 17:44:32.158947 10367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:32.158991 10367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e33f1c70f0c4db58890deed8f12df71, State: Initialized, Role: FOLLOWER
I20260924 17:44:32.159111 10367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e33f1c70f0c4db58890deed8f12df71" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41183 } }
I20260924 17:44:32.159183 10367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:44:32.159226 10367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:44:32.159282 10367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:32.159861 10367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e33f1c70f0c4db58890deed8f12df71" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41183 } }
I20260924 17:44:32.159956 10367 leader_election.cc:302] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e33f1c70f0c4db58890deed8f12df71; no voters:
I20260924 17:44:32.160115 10367 leader_election.cc:288] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:44:32.160147 10372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:32.160257 10372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [term 1 LEADER]: Becoming Leader. State: Replica: 6e33f1c70f0c4db58890deed8f12df71, State: Running, Role: LEADER
I20260924 17:44:32.160341 10372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e33f1c70f0c4db58890deed8f12df71" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41183 } }
I20260924 17:44:32.160391 10367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:44:32.160658 10372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e33f1c70f0c4db58890deed8f12df71. Latest consensus state: current_term: 1 leader_uuid: "6e33f1c70f0c4db58890deed8f12df71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e33f1c70f0c4db58890deed8f12df71" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41183 } } }
I20260924 17:44:32.160665 10373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e33f1c70f0c4db58890deed8f12df71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e33f1c70f0c4db58890deed8f12df71" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41183 } } }
I20260924 17:44:32.160828 10372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:32.160929 10373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:32.161149 10379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:44:32.161705 10379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:44:32.163295 10379 catalog_manager.cc:1347] Generated new cluster ID: c1f46f6a6d2c4c60b7d6cceba569549e
I20260924 17:44:32.163348 10379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:44:32.179957 10379 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:44:32.180500 10379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:44:32.183763 10379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e33f1c70f0c4db58890deed8f12df71: Generated new TSK 0
I20260924 17:44:32.183986 10379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:44:32.185334 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41183
--builtin_ntp_servers=127.1.8.148:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 17:44:32.270186 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.270470 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.270530 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.271783 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.271832 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.271860 10391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:44:32.273272 10391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.273553 10391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.273901 10391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.279934 10396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.279949 10397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.280049 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.280311 10391 server_base.cc:1034] running on GCE node
I20260924 17:44:32.280576 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.280862 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.282015 10391 hybrid_clock.cc:648] HybridClock initialized: now 1790271872281989 us; error 37 us; skew 500 ppm
I20260924 17:44:32.283599 10391 webserver.cc:460] Webserver started at http://127.1.8.129:40301/ using document root <none> and password file <none>
I20260924 17:44:32.283905 10391 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.283977 10391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.284163 10391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.285622 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "d43a75414511425eb2eb7188b443a0e7"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "7cf73d31743f914b90449c10d2874bf6"
server_key_iv: "3269188243077fe24dfa91af76767964"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.286170 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "d43a75414511425eb2eb7188b443a0e7"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "7cf73d31743f914b90449c10d2874bf6"
server_key_iv: "3269188243077fe24dfa91af76767964"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.288251 10391 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:32.289441 10405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.289650 10391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:32.289748 10391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "d43a75414511425eb2eb7188b443a0e7"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "7cf73d31743f914b90449c10d2874bf6"
server_key_iv: "3269188243077fe24dfa91af76767964"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.289835 10391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.307120 10391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.307475 10391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.307628 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.307870 10391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:32.308218 10391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:32.308274 10391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.308317 10391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:32.308346 10391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.314337 10391 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:35075
I20260924 17:44:32.314469 10518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:35075 every 8 connection(s)
I20260924 17:44:32.314811 10391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:44:32.320443 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.315721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56497 (local address 127.1.8.190:41183)
0924 17:44:32.315772 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.315773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.315949 (+ 176us) server_negotiation.cc:407] Connection header received
0924 17:44:32.316412 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.316412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.316425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.316486 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.317166 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.317676 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.318873 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.319068 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.319129 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.320075 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.320075 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.320076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.320080 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.320091 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.320094 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.320134 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.320352 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.320376 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:44:32.320760 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.315579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41183 (local address 127.1.8.129:56497)
0924 17:44:32.315785 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:32.315826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:32.316327 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:32.316549 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:32.316552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:32.316561 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:32.317076 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:32.317081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.317768 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.317769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:32.318790 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:32.318792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.319153 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.319154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:32.319269 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.319940 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:44:32.319940 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:32.319973 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:44:32.319998 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:32.320260 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:32.320266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:32.320269 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:32.320349 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:44:32.320443 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260924 17:44:32.321599 10519 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41183
I20260924 17:44:32.321692 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10391
I20260924 17:44:32.321719 10519 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:32.321779 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:44:32.322098 1058 external_mini_cluster.cc:1428] Setting key 56dd171b5e15bb61ba6eb63af8ad61dc
I20260924 17:44:32.322139 10519 heartbeater.cc:511] Master 127.1.8.190:41183 requested a full tablet report, sending...
I20260924 17:44:32.322795 10330 ts_manager.cc:194] Registered new tserver with Master: d43a75414511425eb2eb7188b443a0e7 (127.1.8.129:35075)
I20260924 17:44:32.323187 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41183
--builtin_ntp_servers=127.1.8.148:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 17:44:32.323540 10330 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.129:56497
W20260924 17:44:32.406504 10522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.406728 10522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.406752 10522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.408093 10522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.408139 10522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.408191 10522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:44:32.409551 10522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.409835 10522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.410156 10522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.415292 10528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.415298 10530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.415337 10527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.415417 10522 server_base.cc:1034] running on GCE node
I20260924 17:44:32.415758 10522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.416028 10522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.417191 10522 hybrid_clock.cc:648] HybridClock initialized: now 1790271872417186 us; error 36 us; skew 500 ppm
I20260924 17:44:32.418550 10522 webserver.cc:460] Webserver started at http://127.1.8.130:44269/ using document root <none> and password file <none>
I20260924 17:44:32.418792 10522 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.418862 10522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.419046 10522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.420325 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "faf1a05bd58a4470b2d85c810fcae136"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "bc3d230fd0c22dc3e4a2f81572ac82c6"
server_key_iv: "6b8701d1930b3dc62809cfa7773adbef"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.420717 10522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "faf1a05bd58a4470b2d85c810fcae136"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "bc3d230fd0c22dc3e4a2f81572ac82c6"
server_key_iv: "6b8701d1930b3dc62809cfa7773adbef"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.422101 10522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:44:32.422942 10536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.423111 10522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.423216 10522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "faf1a05bd58a4470b2d85c810fcae136"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "bc3d230fd0c22dc3e4a2f81572ac82c6"
server_key_iv: "6b8701d1930b3dc62809cfa7773adbef"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.423309 10522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.433281 10522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.433624 10522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.433787 10522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.434038 10522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:32.434373 10522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:32.434465 10522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.434506 10522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:32.434536 10522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.440470 10522 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:35727
I20260924 17:44:32.440603 10649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:35727 every 8 connection(s)
I20260924 17:44:32.440897 10522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:44:32.446038 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.441707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:57977 (local address 127.1.8.190:41183)
0924 17:44:32.441751 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.441753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.442006 (+ 253us) server_negotiation.cc:407] Connection header received
0924 17:44:32.442567 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.442568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.442578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.442630 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.443283 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.443776 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.444746 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.444943 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.444993 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.445777 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.445778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.445778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.445783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.445796 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.445800 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.445842 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.445947 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.445975 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:44:32.446267 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.441691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41183 (local address 127.1.8.130:57977)
0924 17:44:32.441868 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:32.441916 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:32.442468 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:32.442688 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:32.442691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:32.442700 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:32.443209 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:32.443212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.443858 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.443858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:32.444665 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:32.444667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.445020 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.445021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:32.445134 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.445641 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:44:32.445641 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:32.445670 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:44:32.445695 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:32.445860 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:32.445863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:32.445865 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:44:32.445922 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:44:32.446005 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260924 17:44:32.446811 10650 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41183
I20260924 17:44:32.446945 10650 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:32.447141 10650 heartbeater.cc:511] Master 127.1.8.190:41183 requested a full tablet report, sending...
I20260924 17:44:32.447546 10330 ts_manager.cc:194] Registered new tserver with Master: faf1a05bd58a4470b2d85c810fcae136 (127.1.8.130:35727)
I20260924 17:44:32.447933 10330 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.130:57977
I20260924 17:44:32.448191 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10522
I20260924 17:44:32.448287 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:44:32.448490 1058 external_mini_cluster.cc:1428] Setting key 96170925fae807e9ce88d23f5886a8ec
I20260924 17:44:32.449482 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41183
--builtin_ntp_servers=127.1.8.148:44817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 17:44:32.531390 10653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.531647 10653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.531682 10653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.532871 10653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.532953 10653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.532998 10653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:44:32.534385 10653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:44817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.534708 10653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.535034 10653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.540330 10661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.540323 10658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.540341 10659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.540349 10653 server_base.cc:1034] running on GCE node
I20260924 17:44:32.540743 10653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.540964 10653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.542130 10653 hybrid_clock.cc:648] HybridClock initialized: now 1790271872542118 us; error 37 us; skew 500 ppm
I20260924 17:44:32.543839 10653 webserver.cc:460] Webserver started at http://127.1.8.131:41389/ using document root <none> and password file <none>
I20260924 17:44:32.544132 10653 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.544204 10653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.544350 10653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.545845 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "c74878d65a7f4b748ecabae943665ce4"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "5c8973ab0d50d7431ff88b9900c714fb"
server_key_iv: "745346bc8c459487a2bb9b60a80cc1da"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.546221 10653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "c74878d65a7f4b748ecabae943665ce4"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "5c8973ab0d50d7431ff88b9900c714fb"
server_key_iv: "745346bc8c459487a2bb9b60a80cc1da"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.547636 10653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 17:44:32.548389 10667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.548563 10653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:32.548646 10653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "c74878d65a7f4b748ecabae943665ce4"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "5c8973ab0d50d7431ff88b9900c714fb"
server_key_iv: "745346bc8c459487a2bb9b60a80cc1da"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.548704 10653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.560988 10653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.561302 10653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.561415 10653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.561712 10653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:32.562044 10653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:32.562098 10653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.562142 10653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:32.562166 10653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.567940 10653 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:45127
I20260924 17:44:32.568037 10780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:45127 every 8 connection(s)
I20260924 17:44:32.568380 10653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:44:32.573470 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.569189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39497 (local address 127.1.8.190:41183)
0924 17:44:32.569284 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.569286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.569504 (+ 218us) server_negotiation.cc:407] Connection header received
0924 17:44:32.570006 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.570007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.570017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.570072 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.570802 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.571295 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.572236 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.572411 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.572462 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.573231 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.573232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.573233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.573237 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.573250 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.573253 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.573296 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.573382 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.573403 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 17:44:32.573670 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.569150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41183 (local address 127.1.8.131:39497)
0924 17:44:32.569375 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:32.569415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:32.569918 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:32.570144 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:32.570148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:32.570157 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:32.570725 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:32.570729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.571372 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.571373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:32.572158 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:32.572160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:32.572482 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:32.572483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:32.572592 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:32.573098 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:44:32.573099 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:32.573128 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:44:32.573151 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:32.573308 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:32.573313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:32.573315 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:44:32.573369 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 17:44:32.573460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":116,"threads_started":1}
I20260924 17:44:32.574211 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10653
I20260924 17:44:32.574226 10781 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41183
I20260924 17:44:32.574309 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:44:32.574329 10781 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:32.574539 10781 heartbeater.cc:511] Master 127.1.8.190:41183 requested a full tablet report, sending...
I20260924 17:44:32.574546 1058 external_mini_cluster.cc:1428] Setting key 76a35981277afd6935d2a1b32aed3ed1
I20260924 17:44:32.574971 10331 ts_manager.cc:194] Registered new tserver with Master: c74878d65a7f4b748ecabae943665ce4 (127.1.8.131:45127)
I20260924 17:44:32.575457 10331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.131:39497
I20260924 17:44:32.575543 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 17:44:32.577473 10370 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:32.577039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.1.8.190:41183)
0924 17:44:32.577101 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.577103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.577191 (+ 88us) server_negotiation.cc:407] Connection header received
0924 17:44:32.577258 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.577259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.577266 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.577303 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.577398 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41398: BlockingRecv error: recv got EOF from 127.0.0.1:41398 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:44:32.577600 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10391
I20260924 17:44:32.582939 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10522
I20260924 17:44:32.587257 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10653
I20260924 17:44:32.591530 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10300
2026-09-24T17:44:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (590 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T17:44:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:32Z Disabled control of system clock
I20260924 17:44:32.618003 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37235
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:37235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 17:44:32.699827 10792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.700052 10792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.700075 10792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.701210 10792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:32.701251 10792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.701264 10792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.701277 10792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:32.701288 10792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:32.702744 10792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:37235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.702966 10792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.703238 10792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.708506 10800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.708534 10797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.708508 10798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.708848 10792 server_base.cc:1034] running on GCE node
I20260924 17:44:32.709136 10792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.709439 10792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.710593 10792 hybrid_clock.cc:648] HybridClock initialized: now 1790271872710574 us; error 47 us; skew 500 ppm
I20260924 17:44:32.711951 10792 webserver.cc:460] Webserver started at http://127.1.8.190:36691/ using document root <none> and password file <none>
I20260924 17:44:32.712216 10792 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.712291 10792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.712447 10792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.713550 10792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "617f9887fe964958aa404c3868bc8049"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "29963eeab05fe67f27d9ed88e304c6ca"
server_key_iv: "8ad8d0c7662587d2a472575b62d5847a"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.713936 10792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "617f9887fe964958aa404c3868bc8049"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "29963eeab05fe67f27d9ed88e304c6ca"
server_key_iv: "8ad8d0c7662587d2a472575b62d5847a"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.715342 10792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:44:32.716086 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.716249 10792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:44:32.716344 10792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "617f9887fe964958aa404c3868bc8049"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "29963eeab05fe67f27d9ed88e304c6ca"
server_key_iv: "8ad8d0c7662587d2a472575b62d5847a"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.716449 10792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.720345 10792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.720588 10792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.720686 10792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.724474 10792 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:37235
I20260924 17:44:32.724525 10858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:37235 every 8 connection(s)
I20260924 17:44:32.724866 10792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:44:32.725595 10859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:32.728523 10859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049: Bootstrap starting.
I20260924 17:44:32.729161 10859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:32.729480 10859 log.cc:824] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:32.730207 10859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049: No bootstrap required, opened a new log
I20260924 17:44:32.731896 10859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617f9887fe964958aa404c3868bc8049" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37235 } }
I20260924 17:44:32.732079 10859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:32.732128 10859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 617f9887fe964958aa404c3868bc8049, State: Initialized, Role: FOLLOWER
I20260924 17:44:32.732290 10859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617f9887fe964958aa404c3868bc8049" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37235 } }
I20260924 17:44:32.732376 10859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:44:32.732427 10859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:44:32.732479 10859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:32.732637 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10792
I20260924 17:44:32.732717 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:44:32.732904 1058 external_mini_cluster.cc:1428] Setting key 03bc14c09a75cc550df3c7a2c92eece0
I20260924 17:44:32.733281 10859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617f9887fe964958aa404c3868bc8049" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37235 } }
I20260924 17:44:32.733409 10859 leader_election.cc:302] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 617f9887fe964958aa404c3868bc8049; no voters:
I20260924 17:44:32.733582 10859 leader_election.cc:288] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:44:32.733636 10863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:32.733807 10863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [term 1 LEADER]: Becoming Leader. State: Replica: 617f9887fe964958aa404c3868bc8049, State: Running, Role: LEADER
I20260924 17:44:32.733922 10859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:44:32.733942 10863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617f9887fe964958aa404c3868bc8049" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37235 } }
I20260924 17:44:32.734365 10866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "617f9887fe964958aa404c3868bc8049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617f9887fe964958aa404c3868bc8049" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37235 } } }
I20260924 17:44:32.734560 10866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:32.734702 10865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 617f9887fe964958aa404c3868bc8049. Latest consensus state: current_term: 1 leader_uuid: "617f9887fe964958aa404c3868bc8049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "617f9887fe964958aa404c3868bc8049" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37235 } } }
I20260924 17:44:32.734846 10865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:32.734865 10871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:44:32.735514 10871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:44:32.736096 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.733484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.1.8.190:37235)
0924 17:44:32.733854 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.733866 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.733914 (+ 48us) server_negotiation.cc:407] Connection header received
0924 17:44:32.734060 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.734068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.734122 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.734636 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.735294 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.735298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.735299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.735303 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.735344 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.735360 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.735672 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.735817 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.735888 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":133,"threads_started":1}
I20260924 17:44:32.737360 10871 catalog_manager.cc:1347] Generated new cluster ID: 75f6490543f34195ab188a8ae5a7ba05
I20260924 17:44:32.737426 10871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:44:32.746874 10871 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:44:32.747103 10871 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:44:32.752905 10871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 617f9887fe964958aa404c3868bc8049: Generated new TSK 0
I20260924 17:44:32.753158 10871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:44:32.759884 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37235
--builtin_ntp_servers=127.1.8.148:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 17:44:32.842787 10883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.843029 10883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.843052 10883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.844301 10883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.844347 10883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.844373 10883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:44:32.845754 10883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.846037 10883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.846370 10883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.851578 10891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.851586 10888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.851595 10889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.851779 10883 server_base.cc:1034] running on GCE node
I20260924 17:44:32.852135 10883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.852463 10883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.853632 10883 hybrid_clock.cc:648] HybridClock initialized: now 1790271872853614 us; error 33 us; skew 500 ppm
I20260924 17:44:32.855100 10883 webserver.cc:460] Webserver started at http://127.1.8.129:38375/ using document root <none> and password file <none>
I20260924 17:44:32.855378 10883 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.855427 10883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.855607 10883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.856709 10883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "ded565fdd1534db3bb423cfd23faaa9f"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "c8d1116922a58e2bbc37c2f00a87bdc0"
server_key_iv: "54b4cdad0cf2389013e982fd348c914a"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.857076 10883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "ded565fdd1534db3bb423cfd23faaa9f"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "c8d1116922a58e2bbc37c2f00a87bdc0"
server_key_iv: "54b4cdad0cf2389013e982fd348c914a"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.858503 10883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 17:44:32.859323 10897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.859513 10883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:32.859618 10883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "ded565fdd1534db3bb423cfd23faaa9f"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "c8d1116922a58e2bbc37c2f00a87bdc0"
server_key_iv: "54b4cdad0cf2389013e982fd348c914a"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.859697 10883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.864377 10883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.864675 10883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.864823 10883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.865025 10883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:32.865326 10883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:32.865363 10883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.865401 10883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:32.865428 10883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.872282 10883 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:35949
I20260924 17:44:32.872319 11010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:35949 every 8 connection(s)
I20260924 17:44:32.872685 10883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:44:32.875176 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 10883
I20260924 17:44:32.875293 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:44:32.875536 1058 external_mini_cluster.cc:1428] Setting key e2fb3b43088fa401961de8da20ad97ea
I20260924 17:44:32.876081 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.873506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:53939 (local address 127.1.8.190:37235)
0924 17:44:32.873603 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.873606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.873795 (+ 189us) server_negotiation.cc:407] Connection header received
0924 17:44:32.874257 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.874259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.874264 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.874318 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.875778 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.875780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.875780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.875785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.875802 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.875807 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.875888 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.875994 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.876011 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 17:44:32.876309 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.873472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37235 (local address 127.1.8.129:53939)
0924 17:44:32.873674 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:32.873708 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:32.874174 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:32.874375 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:32.874378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:32.874387 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:32.875587 (+ 1200us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:44:32.875588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:32.875633 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:44:32.875672 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:32.875885 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:32.875892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:32.875895 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:32.876005 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 17:44:32.876070 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260924 17:44:32.876629 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37235
--builtin_ntp_servers=127.1.8.148:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 17:44:32.876818 11011 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37235
I20260924 17:44:32.876907 11011 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:32.877075 11011 heartbeater.cc:511] Master 127.1.8.190:37235 requested a full tablet report, sending...
I20260924 17:44:32.877581 10823 ts_manager.cc:194] Registered new tserver with Master: ded565fdd1534db3bb423cfd23faaa9f (127.1.8.129:35949)
W20260924 17:44:32.958845 11014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:32.959064 11014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:32.959087 11014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:32.960345 11014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:32.960394 11014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:32.960456 11014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:44:32.961845 11014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:32.962129 11014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:32.962481 11014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:32.967717 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.967717 11020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:32.967828 11019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:32.967993 11014 server_base.cc:1034] running on GCE node
I20260924 17:44:32.968211 11014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:32.968436 11014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:32.969580 11014 hybrid_clock.cc:648] HybridClock initialized: now 1790271872969569 us; error 37 us; skew 500 ppm
I20260924 17:44:32.974344 11014 webserver.cc:460] Webserver started at http://127.1.8.130:35081/ using document root <none> and password file <none>
I20260924 17:44:32.974668 11014 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:32.974722 11014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:32.974908 11014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:32.976016 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "93f561c0fb964fe3b34a499e6740b2f0"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "56a032f15016b1b517bc4a67253dba52"
server_key_iv: "d4d4f101465312edf4371cefec27d4d8"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.976378 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "93f561c0fb964fe3b34a499e6740b2f0"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "56a032f15016b1b517bc4a67253dba52"
server_key_iv: "d4d4f101465312edf4371cefec27d4d8"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.977756 11014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:44:32.978559 11028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.978767 11014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:32.978865 11014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "93f561c0fb964fe3b34a499e6740b2f0"
format_stamp: "Formatted at 2026-09-24 17:44:32 on dist-test-slave-r7hf"
server_key: "56a032f15016b1b517bc4a67253dba52"
server_key_iv: "d4d4f101465312edf4371cefec27d4d8"
server_key_version: "encryptionkey@0"
I20260924 17:44:32.978955 11014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:32.982328 11014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:32.982638 11014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:32.982782 11014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:32.983004 11014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:32.983309 11014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:32.983341 11014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.983361 11014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:32.983409 11014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:32.989037 11014 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:44857
I20260924 17:44:32.989107 11141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:44857 every 8 connection(s)
I20260924 17:44:32.989470 11014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:44:32.991725 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 11014
I20260924 17:44:32.991837 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:44:32.992061 1058 external_mini_cluster.cc:1428] Setting key 7c8a18db7a3c9b9f3d96604d0f179078
I20260924 17:44:32.992821 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.990215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:38751 (local address 127.1.8.190:37235)
0924 17:44:32.990284 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:32.990286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:32.990546 (+ 260us) server_negotiation.cc:407] Connection header received
0924 17:44:32.991090 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:32.991091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:32.991098 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:32.991158 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:32.992517 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:32.992519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:32.992520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:32.992524 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:32.992542 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:32.992547 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:32.992625 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:32.992734 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 17:44:32.992751 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:44:32.993042 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:32.990189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37235 (local address 127.1.8.130:38751)
0924 17:44:32.990366 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:32.990448 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:32.990975 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:32.991236 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:32.991240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:32.991257 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:32.992356 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:44:32.992356 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:32.992389 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:44:32.992420 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:32.992622 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:32.992627 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:32.992631 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:32.992742 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:44:32.992808 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260924 17:44:32.993150 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37235
--builtin_ntp_servers=127.1.8.148:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 17:44:32.994832 11142 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37235
I20260924 17:44:32.994938 11142 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:32.995115 11142 heartbeater.cc:511] Master 127.1.8.190:37235 requested a full tablet report, sending...
I20260924 17:44:32.995491 10823 ts_manager.cc:194] Registered new tserver with Master: 93f561c0fb964fe3b34a499e6740b2f0 (127.1.8.130:44857)
W20260924 17:44:33.076205 11145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:33.076442 11145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:33.076467 11145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:33.077606 11145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:33.077641 11145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:33.077667 11145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:44:33.079154 11145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:33.079366 11145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:33.079636 11145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:33.084954 11153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:33.084954 11150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:33.084978 11151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:33.085242 11145 server_base.cc:1034] running on GCE node
I20260924 17:44:33.085489 11145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:33.085754 11145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:33.086906 11145 hybrid_clock.cc:648] HybridClock initialized: now 1790271873086886 us; error 35 us; skew 500 ppm
I20260924 17:44:33.088411 11145 webserver.cc:460] Webserver started at http://127.1.8.131:37945/ using document root <none> and password file <none>
I20260924 17:44:33.088663 11145 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:33.088711 11145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:33.088846 11145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:33.089993 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "d6c0d11752c749c7a0a5f4581fa9928f"
format_stamp: "Formatted at 2026-09-24 17:44:33 on dist-test-slave-r7hf"
server_key: "6b02a331e68122b155639ff8c3d264cf"
server_key_iv: "55d640fce309fe893e6b1543f08ba2f7"
server_key_version: "encryptionkey@0"
I20260924 17:44:33.090435 11145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "d6c0d11752c749c7a0a5f4581fa9928f"
format_stamp: "Formatted at 2026-09-24 17:44:33 on dist-test-slave-r7hf"
server_key: "6b02a331e68122b155639ff8c3d264cf"
server_key_iv: "55d640fce309fe893e6b1543f08ba2f7"
server_key_version: "encryptionkey@0"
I20260924 17:44:33.091866 11145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:33.092708 11159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:33.092998 11145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:33.093075 11145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "d6c0d11752c749c7a0a5f4581fa9928f"
format_stamp: "Formatted at 2026-09-24 17:44:33 on dist-test-slave-r7hf"
server_key: "6b02a331e68122b155639ff8c3d264cf"
server_key_iv: "55d640fce309fe893e6b1543f08ba2f7"
server_key_version: "encryptionkey@0"
I20260924 17:44:33.093170 11145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:33.096685 11145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:33.096982 11145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:33.097121 11145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:33.097334 11145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:33.097653 11145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:33.097705 11145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:33.097746 11145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:33.097772 11145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:33.103639 11145 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:39687
I20260924 17:44:33.103677 11272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:39687 every 8 connection(s)
I20260924 17:44:33.104028 11145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:44:33.107200 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:33.104810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:52863 (local address 127.1.8.190:37235)
0924 17:44:33.104874 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:33.104877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:33.105185 (+ 308us) server_negotiation.cc:407] Connection header received
0924 17:44:33.105642 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:33.105644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:33.105649 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:33.105705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:33.106977 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:33.106978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:33.106978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:44:33.106982 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:33.106997 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:44:33.107002 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:33.107068 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:33.107132 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 17:44:33.107146 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:44:33.107388 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:33.104807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37235 (local address 127.1.8.131:52863)
0924 17:44:33.105064 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:33.105098 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:33.105550 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:33.105757 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:33.105760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:33.105769 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:33.106810 (+ 1041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:44:33.106811 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:33.106842 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:44:33.106871 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:33.107067 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:33.107070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:33.107072 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:44:33.107125 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:44:33.107175 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
I20260924 17:44:33.107932 11273 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37235
I20260924 17:44:33.108028 11273 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:33.108142 11273 heartbeater.cc:511] Master 127.1.8.190:37235 requested a full tablet report, sending...
I20260924 17:44:33.108517 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 11145
I20260924 17:44:33.108628 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:44:33.108579 10823 ts_manager.cc:194] Registered new tserver with Master: d6c0d11752c749c7a0a5f4581fa9928f (127.1.8.131:39687)
I20260924 17:44:33.108848 1058 external_mini_cluster.cc:1428] Setting key 4128891bccab089b7f49b5d2e9f84ee5
I20260924 17:44:33.109671 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 17:44:33.111515 10864 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:44:33.111043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.1.8.190:37235)
0924 17:44:33.111084 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:33.111086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:33.111197 (+ 111us) server_negotiation.cc:407] Connection header received
0924 17:44:33.111246 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:33.111246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:33.111252 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:33.111307 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:33.111440 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51256: BlockingRecv error: recv got EOF from 127.0.0.1:51256 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:44:33.111542 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10883
I20260924 17:44:33.116482 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 11014
I20260924 17:44:33.121359 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 11145
I20260924 17:44:33.125897 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 10792
2026-09-24T17:44:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (534 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:44:33 dist-test-slave-r7hf krb5kdc[11283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:44:33 dist-test-slave-r7hf krb5kdc[11283](info): set up 2 sockets
Sep 24 17:44:33 dist-test-slave-r7hf krb5kdc[11283](info): commencing operation
krb5kdc: starting...
W20260924 17:44:34.178874 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:44:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:44:34.309304 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:36737
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:36737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:34.391921 11296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:34.392194 11296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:34.392234 11296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:34.393440 11296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:44:34.393514 11296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:34.393538 11296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:34.393559 11296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:44:34.393622 11296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:44:34.395190 11296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:36737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:36737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:34.395483 11296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:34.395715 11296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:34.400717 11301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:34.400813 11302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:34.400853 11296 server_base.cc:1034] running on GCE node
W20260924 17:44:34.400799 11304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:34.401234 11296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:34.401510 11296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:34.402688 11296 hybrid_clock.cc:648] HybridClock initialized: now 1790271874402662 us; error 36 us; skew 500 ppm
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:34.405203 11296 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:44:34.405649 11296 webserver.cc:460] Webserver started at http://127.1.8.190:39893/ using document root <none> and password file <none>
I20260924 17:44:34.405857 11296 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:34.405900 11296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:34.405992 11296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:34.407145 11296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "34210c5c5c594324a176b173fea0d9f6"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "4da755992e8ed2cf9d143238de7964b7"
server_key_iv: "345feab71f521a7c72896a20ff9a8281"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.407490 11296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "34210c5c5c594324a176b173fea0d9f6"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "4da755992e8ed2cf9d143238de7964b7"
server_key_iv: "345feab71f521a7c72896a20ff9a8281"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.408815 11296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:44:34.409534 11311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.409734 11296 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:44:34.409823 11296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "34210c5c5c594324a176b173fea0d9f6"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "4da755992e8ed2cf9d143238de7964b7"
server_key_iv: "345feab71f521a7c72896a20ff9a8281"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.409895 11296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:34.427374 11296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:34.429325 11296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:34.429446 11296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:34.433410 11296 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:36737
I20260924 17:44:34.433519 11363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:36737 every 8 connection(s)
I20260924 17:44:34.433847 11296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:44:34.434525 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 11296
I20260924 17:44:34.434666 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:44:34.434911 1058 external_mini_cluster.cc:1428] Setting key 678d7fb304a4f8e5b73e1812f4534e9d
I20260924 17:44:34.435200 11364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:34.438030 11364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6: Bootstrap starting.
I20260924 17:44:34.438733 11364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:34.439035 11364 log.cc:824] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:34.439743 11364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6: No bootstrap required, opened a new log
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:44:34.441874 11364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34210c5c5c594324a176b173fea0d9f6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36737 } }
I20260924 17:44:34.442036 11364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:34.442063 11364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34210c5c5c594324a176b173fea0d9f6, State: Initialized, Role: FOLLOWER
I20260924 17:44:34.442216 11364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34210c5c5c594324a176b173fea0d9f6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36737 } }
I20260924 17:44:34.442303 11364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:44:34.442327 11364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:44:34.442421 11364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:34.443037 11364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34210c5c5c594324a176b173fea0d9f6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36737 } }
I20260924 17:44:34.443151 11364 leader_election.cc:302] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34210c5c5c594324a176b173fea0d9f6; no voters:
I20260924 17:44:34.443369 11364 leader_election.cc:288] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:44:34.443399 11369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:34.443596 11369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [term 1 LEADER]: Becoming Leader. State: Replica: 34210c5c5c594324a176b173fea0d9f6, State: Running, Role: LEADER
I20260924 17:44:34.443707 11369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34210c5c5c594324a176b173fea0d9f6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36737 } }
I20260924 17:44:34.443737 11364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:44:34.444199 11371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34210c5c5c594324a176b173fea0d9f6. Latest consensus state: current_term: 1 leader_uuid: "34210c5c5c594324a176b173fea0d9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34210c5c5c594324a176b173fea0d9f6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36737 } } }
I20260924 17:44:34.444303 11371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:34.444521 11374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:44:34.444199 11370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34210c5c5c594324a176b173fea0d9f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34210c5c5c594324a176b173fea0d9f6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 36737 } } }
I20260924 17:44:34.444859 11370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6 [sys.catalog]: This master's current role is: LEADER
I20260924 17:44:34.445161 11374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:44:34.446520 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.435447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.1.8.190:36737)
0924 17:44:34.435653 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:34.435664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:34.435704 (+ 40us) server_negotiation.cc:407] Connection header received
0924 17:44:34.436329 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:34.436336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:34.436400 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:34.436555 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:34.437199 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.438112 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.438974 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.439470 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.439597 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.442518 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:34.442523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:34.442523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:34.442539 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:34.444696 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.445200 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.445203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.445204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.445320 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.445559 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.445560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.445560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.445829 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:34.446000 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:34.446110 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:44:34.446248 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260924 17:44:34.454630 11374 catalog_manager.cc:1347] Generated new cluster ID: ab9fe72511a84f208db01a2a9de95c1c
I20260924 17:44:34.454708 11374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:44:34.459301 11374 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:44:34.459836 11374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:44:34.465232 11374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6: Generated new TSK 0
I20260924 17:44:34.465459 11374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:44:34.497769 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36737
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:34.583629 11390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:34.583868 11390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:34.583894 11390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:34.585172 11390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:34.585228 11390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:34.585258 11390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:44:34.587039 11390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:34.587311 11390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:34.587577 11390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:34.592895 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:34.592916 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:34.592916 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:34.593361 11390 server_base.cc:1034] running on GCE node
I20260924 17:44:34.593598 11390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:34.593813 11390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:34.594992 11390 hybrid_clock.cc:648] HybridClock initialized: now 1790271874594975 us; error 46 us; skew 500 ppm
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:34.597690 11390 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:44:34.598163 11390 webserver.cc:460] Webserver started at http://127.1.8.129:33505/ using document root <none> and password file <none>
I20260924 17:44:34.598462 11390 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:34.598543 11390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:34.598704 11390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:34.599809 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "aefe6cb4fdcb45e18f98b3b43107e72b"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "8f3087b57caf7a31de711ca5b44a47a4"
server_key_iv: "ea049b63a05cbea1ac68dace94eb0637"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.600193 11390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "aefe6cb4fdcb45e18f98b3b43107e72b"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "8f3087b57caf7a31de711ca5b44a47a4"
server_key_iv: "ea049b63a05cbea1ac68dace94eb0637"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.601612 11390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:44:34.602465 11405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.602674 11390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:34.602799 11390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "aefe6cb4fdcb45e18f98b3b43107e72b"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "8f3087b57caf7a31de711ca5b44a47a4"
server_key_iv: "ea049b63a05cbea1ac68dace94eb0637"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.602883 11390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:34.631456 11390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:34.633719 11390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:34.633838 11390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:34.634132 11390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:34.634521 11390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:34.634562 11390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.634585 11390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:34.634616 11390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.640597 11390 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:44105
I20260924 17:44:34.640640 11518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:44105 every 8 connection(s)
I20260924 17:44:34.641031 11390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:44:34.643388 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 11390
I20260924 17:44:34.643484 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:44:34.643733 1058 external_mini_cluster.cc:1428] Setting key a51aad9f5685501bf45b368f9e606d8e
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:44:34.652078 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.641967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:53737 (local address 127.1.8.190:36737)
0924 17:44:34.642053 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:34.642055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:34.642186 (+ 131us) server_negotiation.cc:407] Connection header received
0924 17:44:34.642702 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:34.642703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:34.642713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:34.642771 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:34.644034 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.644804 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.646214 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.646515 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.646600 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.648984 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:34.648987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:34.648987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:34.649000 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:34.650552 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.650991 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.650992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.650993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.651055 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.651330 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.651331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.651331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.651457 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:34.651536 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:34.651870 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 17:44:34.651974 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 17:44:34.652282 11521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.641889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:36737 (local address 127.1.8.129:53737)
0924 17:44:34.642052 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:34.642093 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:34.642613 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:34.642841 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:34.642844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:34.643583 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:34.643904 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:34.643912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.644897 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.644898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:34.646085 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:34.646088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.646606 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.646607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:34.646767 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.647378 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:34.647387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:34.648866 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:34.650654 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:34.650662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:34.650664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:34.650907 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:34.651130 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:34.651131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:34.651132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:34.651259 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:34.651526 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:34.651532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:34.651716 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 17:44:34.651838 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 17:44:34.652010 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260924 17:44:34.652828 11519 heartbeater.cc:347] Connected to a master server at 127.1.8.190:36737
I20260924 17:44:34.652954 11519 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:34.653146 11519 heartbeater.cc:511] Master 127.1.8.190:36737 requested a full tablet report, sending...
I20260924 17:44:34.653831 11328 ts_manager.cc:194] Registered new tserver with Master: aefe6cb4fdcb45e18f98b3b43107e72b (127.1.8.129:44105)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
I20260924 17:44:34.654652 11328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:53737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:44:34.669852 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36737
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:34.752846 11524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:34.753059 11524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:34.753083 11524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:34.754184 11524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:34.754218 11524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:34.754244 11524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:44:34.755764 11524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:34.755956 11524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:34.756145 11524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:34.761380 11532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:34.761380 11530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:34.761428 11524 server_base.cc:1034] running on GCE node
W20260924 17:44:34.761379 11529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:34.761906 11524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:34.762173 11524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:34.763316 11524 hybrid_clock.cc:648] HybridClock initialized: now 1790271874763283 us; error 52 us; skew 500 ppm
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:34.765872 11524 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:44:34.766258 11524 webserver.cc:460] Webserver started at http://127.1.8.130:42157/ using document root <none> and password file <none>
I20260924 17:44:34.766526 11524 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:34.766577 11524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:34.766670 11524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:34.767707 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "e4fbe828de564e368dbcaa67bc2f50eb"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "f44da3ce23f5adf0128ee55742b5e4ec"
server_key_iv: "984513e69b2208fdc63aa86460f5e028"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.768020 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "e4fbe828de564e368dbcaa67bc2f50eb"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "f44da3ce23f5adf0128ee55742b5e4ec"
server_key_iv: "984513e69b2208fdc63aa86460f5e028"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.769367 11524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:44:34.770080 11539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.770254 11524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:44:34.770344 11524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "e4fbe828de564e368dbcaa67bc2f50eb"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "f44da3ce23f5adf0128ee55742b5e4ec"
server_key_iv: "984513e69b2208fdc63aa86460f5e028"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.770445 11524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:34.805017 11524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:34.807510 11524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:34.807675 11524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:34.807946 11524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:34.808282 11524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:34.808318 11524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.808360 11524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:34.808388 11524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.814599 11524 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:44069
I20260924 17:44:34.814642 11652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:44069 every 8 connection(s)
I20260924 17:44:34.814990 11524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:44:34.815446 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 11524
I20260924 17:44:34.815578 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:44:34.815830 1058 external_mini_cluster.cc:1428] Setting key de6789e409df87da38a4cf7d689fcec6
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:44:34.827529 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.816421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:51317 (local address 127.1.8.190:36737)
0924 17:44:34.816463 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:34.816465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:34.816908 (+ 443us) server_negotiation.cc:407] Connection header received
0924 17:44:34.817665 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:34.817666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:34.817676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:34.817716 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:34.818945 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.819606 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.821330 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.821773 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.821850 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.824537 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:34.824538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:34.824538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:34.824547 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:34.826151 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.826598 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.826599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.826599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.826649 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.826892 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.826892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.826892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.826998 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:34.827080 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:34.827385 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 17:44:34.827460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:44:34.827734 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.816416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:36737 (local address 127.1.8.130:51317)
0924 17:44:34.816749 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:34.816804 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:34.817554 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:34.817801 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:34.817805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:34.818641 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:34.818846 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:34.818851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.819833 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.819835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:34.821236 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:34.821247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.821882 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.821883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:34.822065 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.822707 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:34.822718 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:34.824431 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:34.826244 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:34.826250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:34.826252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:34.826541 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:34.826715 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:34.826716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:34.826716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:34.826842 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:34.827087 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:34.827093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:34.827243 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 17:44:34.827341 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 17:44:34.827480 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
I20260924 17:44:34.828265 11653 heartbeater.cc:347] Connected to a master server at 127.1.8.190:36737
I20260924 17:44:34.828387 11653 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:34.828606 11653 heartbeater.cc:511] Master 127.1.8.190:36737 requested a full tablet report, sending...
I20260924 17:44:34.829052 11328 ts_manager.cc:194] Registered new tserver with Master: e4fbe828de564e368dbcaa67bc2f50eb (127.1.8.130:44069)
I20260924 17:44:34.829438 11328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:51317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:44:34.845934 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36737
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:44:34.930718 11658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:44:34.930951 11658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:44:34.930979 11658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:44:34.932137 11658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:44:34.932190 11658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:44:34.932221 11658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:44:34.933732 11658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:36737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:44:34.933955 11658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:44:34.934145 11658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:44:34.939389 11666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:34.939432 11664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:44:34.939390 11663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:44:34.939805 11658 server_base.cc:1034] running on GCE node
I20260924 17:44:34.940017 11658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:44:34.940266 11658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:44:34.941438 11658 hybrid_clock.cc:648] HybridClock initialized: now 1790271874941420 us; error 24 us; skew 500 ppm
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:44:34.944377 11658 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:44:34.944998 11658 webserver.cc:460] Webserver started at http://127.1.8.131:33665/ using document root <none> and password file <none>
I20260924 17:44:34.945314 11658 fs_manager.cc:359] Metadata directory not provided
I20260924 17:44:34.945366 11658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:44:34.945531 11658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:44:34.946772 11658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "e409c81f65e541f487ef18faceb574e6"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "f32a9cff9e607465c9db6fec8408f5e3"
server_key_iv: "6ff1846c333d02c448cfc68068dece21"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.947150 11658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "e409c81f65e541f487ef18faceb574e6"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "f32a9cff9e607465c9db6fec8408f5e3"
server_key_iv: "6ff1846c333d02c448cfc68068dece21"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.948583 11658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:44:34.949426 11673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.949602 11658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:44:34.949705 11658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "e409c81f65e541f487ef18faceb574e6"
format_stamp: "Formatted at 2026-09-24 17:44:34 on dist-test-slave-r7hf"
server_key: "f32a9cff9e607465c9db6fec8408f5e3"
server_key_iv: "6ff1846c333d02c448cfc68068dece21"
server_key_version: "encryptionkey@0"
I20260924 17:44:34.949803 11658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:44:34.974766 11658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:44:34.976639 11658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:44:34.976812 11658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:44:34.977061 11658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:44:34.977378 11658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:44:34.977434 11658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.977480 11658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:44:34.977505 11658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:44:34.983520 11658 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:44329
I20260924 17:44:34.983559 11786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:44329 every 8 connection(s)
I20260924 17:44:34.983894 11658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:44:34 dist-test-slave-r7hf krb5kdc[11283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:44:34.991664 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 11658
I20260924 17:44:34.991770 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:44:34.991977 1058 external_mini_cluster.cc:1428] Setting key d900b6d5b44a5e4fe3f145c6ae22dfc9
I20260924 17:44:34.993887 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.984624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:46693 (local address 127.1.8.190:36737)
0924 17:44:34.984691 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:34.984693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:34.984896 (+ 203us) server_negotiation.cc:407] Connection header received
0924 17:44:34.985517 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:34.985518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:34.985527 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:34.985591 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:34.986578 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.987073 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.988072 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.988276 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.988345 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.990722 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:34.990724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:34.990724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:34.990733 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:34.992522 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.993079 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.993080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.993081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.993135 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:34.993334 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:34.993334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:34.993335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:34.993449 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:34.993501 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:34.993732 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 17:44:34.993823 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:44:34.994055 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:34.984589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:36737 (local address 127.1.8.131:46693)
0924 17:44:34.984767 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:34.984809 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:34.985418 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:34.985674 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:34.985678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:34.986262 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:44:34.986498 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:34.986502 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.987156 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.987157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:34.987954 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:34.987956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:34.988360 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:34.988361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:34.988473 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:34.989023 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:44:34.989032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:44:34.990613 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:44:34.992652 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:34.992661 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:34.992663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:34.993006 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:34.993180 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:44:34.993180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:44:34.993181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:44:34.993276 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:44:34.993492 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:44:34.993495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:44:34.993621 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 17:44:34.993702 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:44:34.993831 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260924 17:44:34.994536 11787 heartbeater.cc:347] Connected to a master server at 127.1.8.190:36737
I20260924 17:44:34.994627 11787 heartbeater.cc:464] Registering TS with master...
I20260924 17:44:34.994827 11787 heartbeater.cc:511] Master 127.1.8.190:36737 requested a full tablet report, sending...
I20260924 17:44:34.995240 11328 ts_manager.cc:194] Registered new tserver with Master: e409c81f65e541f487ef18faceb574e6 (127.1.8.131:44329)
I20260924 17:44:34.995640 11328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:46693
I20260924 17:44:35.003867 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:44:35.011824 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.005368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.1.8.190:36737)
0924 17:44:35.005414 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.005416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.005603 (+ 187us) server_negotiation.cc:407] Connection header received
0924 17:44:35.005642 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.005642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.005652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.005700 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:44:35.006246 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.006841 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.007579 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.007845 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.007913 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.009107 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:44:35.009108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:44:35.009109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:44:35.009115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:44:35.010622 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:35.011063 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:35.011064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:35.011064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:35.011118 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:44:35.011307 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:44:35.011308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:44:35.011308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:44:35.011456 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:44:35.011541 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.011622 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.011685 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:44:35.013940 11328 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:44:35.016104 11328 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:44:35.022850 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.018539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.1.8.131:44329)
0924 17:44:35.018684 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.018689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.018828 (+ 139us) server_negotiation.cc:407] Connection header received
0924 17:44:35.018938 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.018940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.018958 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.019036 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:35.019336 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.020489 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.021724 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.022479 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.022560 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.022620 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.022738 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.022781 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260924 17:44:35.022863 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.018436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:44329 (local address 127.0.0.1:50568)
0924 17:44:35.018725 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:35.018766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:35.018846 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:35.019162 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:35.019166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:35.019168 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:35.019248 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.019250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.020600 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.020601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.021656 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.021658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.022560 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.022561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.022638 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.022652 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 17:44:35.022730 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:44:35.022784 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260924 17:44:35.023309 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.018643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:44069 (local address 127.0.0.1:46998)
0924 17:44:35.019019 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:35.019030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:35.019082 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:35.019355 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:35.019356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:35.019358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:35.019408 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.019410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.020275 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.020276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.021556 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.021558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.023074 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.023075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.023135 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.023138 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:35.023185 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 17:44:35.023225 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260924 17:44:35.023607 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.018441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44105 (local address 127.0.0.1:33194)
0924 17:44:35.018868 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:35.018878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:35.018940 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:35.019384 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:35.019385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:35.019387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:35.019431 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.019432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.020654 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.020655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.021565 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.021567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.023378 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.023379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.023461 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.023465 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:35.023514 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:44:35.023546 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":88,"threads_started":1}
I20260924 17:44:35.023622 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.018854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.1.8.130:44069)
0924 17:44:35.019012 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.019017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.019075 (+ 58us) server_negotiation.cc:407] Connection header received
0924 17:44:35.019149 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.019151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.019165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.019254 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:35.019490 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.020168 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.021650 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.022480 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.022560 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.022620 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.023491 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.023540 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260924 17:44:35.023641 11721 tablet_service.cc:1467] Processing CreateTablet for tablet b98f41cdcf044fd1add67f2c06db6407 (DEFAULT_TABLE table=test-table [id=b629bdb391d14b7993001cd3799b8f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:35.023834 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.018552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.1.8.129:44105)
0924 17:44:35.018774 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.018778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.019001 (+ 223us) server_negotiation.cc:407] Connection header received
0924 17:44:35.019038 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.019040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.019055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.019305 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:35.019506 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.020553 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.022488 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.023233 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.023293 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.023345 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.023712 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.023760 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260924 17:44:35.024050 11721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f41cdcf044fd1add67f2c06db6407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:35.026757 11804 tablet_bootstrap.cc:492] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Bootstrap starting.
I20260924 17:44:35.027170 11587 tablet_service.cc:1467] Processing CreateTablet for tablet b98f41cdcf044fd1add67f2c06db6407 (DEFAULT_TABLE table=test-table [id=b629bdb391d14b7993001cd3799b8f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:35.027436 11804 tablet_bootstrap.cc:654] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:35.027513 11587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f41cdcf044fd1add67f2c06db6407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:35.027976 11804 log.cc:824] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:35.028725 11804 tablet_bootstrap.cc:492] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: No bootstrap required, opened a new log
I20260924 17:44:35.028784 11804 ts_tablet_manager.cc:1397] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:35.030303 11806 tablet_bootstrap.cc:492] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Bootstrap starting.
I20260924 17:44:35.030242 11804 raft_consensus.cc:348] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.030566 11804 raft_consensus.cc:374] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:35.030620 11804 raft_consensus.cc:729] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e409c81f65e541f487ef18faceb574e6, State: Initialized, Role: FOLLOWER
I20260924 17:44:35.030778 11804 consensus_queue.cc:260] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.030776 11453 tablet_service.cc:1467] Processing CreateTablet for tablet b98f41cdcf044fd1add67f2c06db6407 (DEFAULT_TABLE table=test-table [id=b629bdb391d14b7993001cd3799b8f8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:44:35.031044 11806 tablet_bootstrap.cc:654] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:35.031042 11804 ts_tablet_manager.cc:1428] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:35.031090 11453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98f41cdcf044fd1add67f2c06db6407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:44:35.031167 11787 heartbeater.cc:503] Master 127.1.8.190:36737 was elected leader, sending a full tablet report...
I20260924 17:44:35.031600 11806 log.cc:824] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:35.032366 11806 tablet_bootstrap.cc:492] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: No bootstrap required, opened a new log
I20260924 17:44:35.032433 11806 ts_tablet_manager.cc:1397] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:35.033783 11806 raft_consensus.cc:348] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.033954 11809 tablet_bootstrap.cc:492] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: Bootstrap starting.
I20260924 17:44:35.034004 11806 raft_consensus.cc:374] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:35.034041 11806 raft_consensus.cc:729] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fbe828de564e368dbcaa67bc2f50eb, State: Initialized, Role: FOLLOWER
I20260924 17:44:35.034155 11806 consensus_queue.cc:260] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.034497 11806 ts_tablet_manager.cc:1428] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:35.034502 11653 heartbeater.cc:503] Master 127.1.8.190:36737 was elected leader, sending a full tablet report...
I20260924 17:44:35.034657 11809 tablet_bootstrap.cc:654] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: Neither blocks nor log segments found. Creating new log.
I20260924 17:44:35.035286 11809 log.cc:824] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:44:35.036018 11809 tablet_bootstrap.cc:492] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: No bootstrap required, opened a new log
I20260924 17:44:35.036082 11809 ts_tablet_manager.cc:1397] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:35.037405 11809 raft_consensus.cc:348] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.037539 11809 raft_consensus.cc:374] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:44:35.037560 11809 raft_consensus.cc:729] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aefe6cb4fdcb45e18f98b3b43107e72b, State: Initialized, Role: FOLLOWER
I20260924 17:44:35.037684 11809 consensus_queue.cc:260] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.037930 11809 ts_tablet_manager.cc:1428] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:44:35.037992 11519 heartbeater.cc:503] Master 127.1.8.190:36737 was elected leader, sending a full tablet report...
W20260924 17:44:35.065562 11654 tablet.cc:2369] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:44:35.141846 11520 tablet.cc:2369] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:44:35.185583 11807 raft_consensus.cc:482] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:44:35.185765 11807 raft_consensus.cc:504] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.186178 11807 leader_election.cc:288] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aefe6cb4fdcb45e18f98b3b43107e72b (127.1.8.129:44105), e4fbe828de564e368dbcaa67bc2f50eb (127.1.8.130:44069)
I20260924 17:44:35.189109 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.186452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44105 (local address 127.1.8.131:39565)
0924 17:44:35.186506 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:35.186522 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:35.186600 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:35.186804 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:35.186805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:35.186806 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:35.186876 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.186878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.187543 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.187544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.188277 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.188279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.188834 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.188835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.188923 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.188926 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:44:35.188993 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 17:44:35.189035 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 17:44:35.189109 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.186510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39565 (local address 127.1.8.129:44105)
0924 17:44:35.186567 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.186570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.186583 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:44:35.186681 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.186683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.186695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.186783 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:35.186968 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.187450 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.188377 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.188747 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.188835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.188845 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.189001 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.189035 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:44:35.189420 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.186487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:33547 (local address 127.1.8.130:44069)
0924 17:44:35.186535 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.186537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.186772 (+ 235us) server_negotiation.cc:407] Connection header received
0924 17:44:35.186825 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.186827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.186839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.186931 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:44:35.187094 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.187589 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.188598 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.189010 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.189103 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.189113 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.189292 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.189339 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:44:35.189472 11473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f41cdcf044fd1add67f2c06db6407" candidate_uuid: "e409c81f65e541f487ef18faceb574e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" is_pre_election: true
I20260924 17:44:35.189633 11473 raft_consensus.cc:2454] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e409c81f65e541f487ef18faceb574e6 in term 0.
I20260924 17:44:35.189661 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.186452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:44069 (local address 127.1.8.131:33547)
0924 17:44:35.186701 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:35.186708 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 17:44:35.186769 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:44:35.186933 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:44:35.186933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:44:35.186935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:44:35.187020 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.187021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.187682 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.187683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.188496 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:44:35.188497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.189103 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.189104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:44:35.189185 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.189189 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:44:35.189520 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 17:44:35.189571 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":53,"thread_start_us":144,"threads_started":1}
I20260924 17:44:35.189918 11674 leader_election.cc:302] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aefe6cb4fdcb45e18f98b3b43107e72b, e409c81f65e541f487ef18faceb574e6; no voters:
I20260924 17:44:35.190021 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f41cdcf044fd1add67f2c06db6407" candidate_uuid: "e409c81f65e541f487ef18faceb574e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" is_pre_election: true
I20260924 17:44:35.190078 11807 raft_consensus.cc:2781] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:44:35.190161 11807 raft_consensus.cc:482] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:44:35.190191 11807 raft_consensus.cc:3037] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:35.190200 11607 raft_consensus.cc:2454] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e409c81f65e541f487ef18faceb574e6 in term 0.
I20260924 17:44:35.191035 11807 raft_consensus.cc:504] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.191195 11807 leader_election.cc:288] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [CANDIDATE]: Term 1 election: Requested vote from peers aefe6cb4fdcb45e18f98b3b43107e72b (127.1.8.129:44105), e4fbe828de564e368dbcaa67bc2f50eb (127.1.8.130:44069)
I20260924 17:44:35.191397 11607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f41cdcf044fd1add67f2c06db6407" candidate_uuid: "e409c81f65e541f487ef18faceb574e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fbe828de564e368dbcaa67bc2f50eb"
I20260924 17:44:35.191396 11473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98f41cdcf044fd1add67f2c06db6407" candidate_uuid: "e409c81f65e541f487ef18faceb574e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b"
I20260924 17:44:35.191480 11607 raft_consensus.cc:3037] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:35.191481 11473 raft_consensus.cc:3037] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:44:35.192425 11607 raft_consensus.cc:2454] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e409c81f65e541f487ef18faceb574e6 in term 1.
I20260924 17:44:35.192425 11473 raft_consensus.cc:2454] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e409c81f65e541f487ef18faceb574e6 in term 1.
I20260924 17:44:35.192598 11674 leader_election.cc:302] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aefe6cb4fdcb45e18f98b3b43107e72b, e409c81f65e541f487ef18faceb574e6; no voters:
I20260924 17:44:35.192713 11807 raft_consensus.cc:2781] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:44:35.192919 11807 raft_consensus.cc:686] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 1 LEADER]: Becoming Leader. State: Replica: e409c81f65e541f487ef18faceb574e6, State: Running, Role: LEADER
I20260924 17:44:35.193022 11807 consensus_queue.cc:237] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } }
I20260924 17:44:35.193938 11328 catalog_manager.cc:5495] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e409c81f65e541f487ef18faceb574e6 (127.1.8.131). New cstate: current_term: 1 leader_uuid: "e409c81f65e541f487ef18faceb574e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e409c81f65e541f487ef18faceb574e6" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 44329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 } health_report { overall_health: UNKNOWN } } }
I20260924 17:44:35.203555 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.201198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.1.8.131:44329)
0924 17:44:35.201239 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:44:35.201241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:44:35.201259 (+ 18us) server_negotiation.cc:407] Connection header received
0924 17:44:35.201314 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:44:35.201315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:44:35.201333 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:44:35.201410 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:44:35.201569 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.202077 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.202858 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:44:35.203059 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:44:35.203115 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:44:35.203251 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:44:35.203354 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:44:35.203441 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 17:44:35.203459 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 17:44:35.203476 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:44:35.205567 11473 raft_consensus.cc:1261] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 1 FOLLOWER]: Refusing update from remote peer e409c81f65e541f487ef18faceb574e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:35.205616 11607 raft_consensus.cc:1261] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 1 FOLLOWER]: Refusing update from remote peer e409c81f65e541f487ef18faceb574e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:44:35.205978 11814 consensus_queue.cc:1035] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fbe828de564e368dbcaa67bc2f50eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:35.206115 11807 consensus_queue.cc:1035] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aefe6cb4fdcb45e18f98b3b43107e72b" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:44:35.208571 11817 mvcc.cc:205] Tried to move back new op lower bound from 7332953600838844416 to 7332953600791375872. Current Snapshot: MvccSnapshot[applied={T|T < 7332953600838844416}]
I20260924 17:44:35.208985 11818 mvcc.cc:205] Tried to move back new op lower bound from 7332953600838844416 to 7332953600791375872. Current Snapshot: MvccSnapshot[applied={T|T < 7332953600838844416}]
I20260924 17:44:35.209750 11819 mvcc.cc:205] Tried to move back new op lower bound from 7332953600838844416 to 7332953600791375872. Current Snapshot: MvccSnapshot[applied={T|T < 7332953600838844416}]
I20260924 17:44:35.211046 11327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:35.211161 11327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:44:35.211990 11327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34210c5c5c594324a176b173fea0d9f6: Sending DeleteTablet for 3 replicas of tablet b98f41cdcf044fd1add67f2c06db6407
I20260924 17:44:35.212327 11453 tablet_service.cc:1514] Processing DeleteTablet for tablet b98f41cdcf044fd1add67f2c06db6407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:35 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:33194
I20260924 17:44:35.212366 11721 tablet_service.cc:1514] Processing DeleteTablet for tablet b98f41cdcf044fd1add67f2c06db6407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:35 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:50568
I20260924 17:44:35.212563 11587 tablet_service.cc:1514] Processing DeleteTablet for tablet b98f41cdcf044fd1add67f2c06db6407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:44:35 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:46998
I20260924 17:44:35.212692 11827 tablet_replica.cc:330] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b: stopping tablet replica
I20260924 17:44:35.212800 11827 raft_consensus.cc:2229] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:35.212839 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 11390
I20260924 17:44:35.212921 11829 tablet_replica.cc:330] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: stopping tablet replica
I20260924 17:44:35.212921 11827 raft_consensus.cc:2258] T b98f41cdcf044fd1add67f2c06db6407 P aefe6cb4fdcb45e18f98b3b43107e72b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:35.213011 11829 raft_consensus.cc:2229] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:44:35.213073 11828 tablet_replica.cc:330] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: stopping tablet replica
I20260924 17:44:35.213105 11829 raft_consensus.cc:2258] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:35.213163 11828 raft_consensus.cc:2229] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:44:35.213613 11828 raft_consensus.cc:2258] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:44:35.213992 11829 ts_tablet_manager.cc:1905] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:44:35.215862 11829 ts_tablet_manager.cc:1918] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:35.215970 11829 log.cc:1196] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/b98f41cdcf044fd1add67f2c06db6407
I20260924 17:44:35.216255 11829 ts_tablet_manager.cc:1939] T b98f41cdcf044fd1add67f2c06db6407 P e4fbe828de564e368dbcaa67bc2f50eb: Deleting consensus metadata
I20260924 17:44:35.216714 11315 catalog_manager.cc:4841] TS e4fbe828de564e368dbcaa67bc2f50eb (127.1.8.130:44069): tablet b98f41cdcf044fd1add67f2c06db6407 (table test-table [id=b629bdb391d14b7993001cd3799b8f8d]) successfully deleted
I20260924 17:44:35.219362 11828 ts_tablet_manager.cc:1905] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:44:35.220556 11312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:44105 (error 108)
I20260924 17:44:35.220939 11828 ts_tablet_manager.cc:1918] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:44:35.221045 11828 log.cc:1196] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/b98f41cdcf044fd1add67f2c06db6407
I20260924 17:44:35.221153 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:44:35.221026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44105 (local address 127.0.0.1:33204)
0924 17:44:35.221081 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 17:44:35.221118 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:44105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 17:44:35.221295 11312 catalog_manager.cc:4584] TS aefe6cb4fdcb45e18f98b3b43107e72b (127.1.8.129:44105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b98f41cdcf044fd1add67f2c06db6407: Network error: Client connection negotiation failed: client connection to 127.1.8.129:44105: connect: Connection refused (error 111)
I20260924 17:44:35.221410 11828 ts_tablet_manager.cc:1939] T b98f41cdcf044fd1add67f2c06db6407 P e409c81f65e541f487ef18faceb574e6: Deleting consensus metadata
I20260924 17:44:35.221679 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 11524
I20260924 17:44:35.222352 11315 catalog_manager.cc:4841] TS e409c81f65e541f487ef18faceb574e6 (127.1.8.131:44329): tablet b98f41cdcf044fd1add67f2c06db6407 (table test-table [id=b629bdb391d14b7993001cd3799b8f8d]) successfully deleted
I20260924 17:44:35.226994 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 11658
I20260924 17:44:35.233433 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 11296
2026-09-24T17:44:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2108 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:44:35 dist-test-slave-r7hf krb5kdc[11833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:44:35 dist-test-slave-r7hf krb5kdc[11833](info): set up 2 sockets
Sep 24 17:44:35 dist-test-slave-r7hf krb5kdc[11833](info): commencing operation
krb5kdc: starting...
W20260924 17:44:36.278896 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:44:36 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:44:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:44:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/rangeradmin_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/rangeradmin_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/rangerlookup_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/rangerlookup_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
I20260924 17:44:36.453403 1058 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkSMFuh/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/postgres -l logfile start
2026-09-24 17:44:39.264 UTC [11863] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 17:44:39.264 UTC [11863] LOG: listening on IPv4 address "127.1.8.148", port 41593
2026-09-24 17:44:39.267 UTC [11863] LOG: listening on Unix socket "/tmp/.s.PGSQL.41593"
2026-09-24 17:44:39.290 UTC [11866] LOG: database system was shut down at 2026-09-24 17:44:37 UTC
2026-09-24 17:44:39.294 UTC [11863] LOG: database system is ready to accept connections
I20260924 17:44:40.246704 1058 mini_postgres.cc:96] Postgres bound to 41593
2026-09-24 17:44:40.252 UTC [11874] FATAL: database "slave" does not exist
127.1.8.148:41593 - accepting connections
I20260924 17:44:40.253577 1058 mini_ranger.cc:161] Starting Ranger...
I20260924 17:44:40.269173 1058 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 17:44:40.822734 1058 mini_ranger.cc:87] Created ranger Postgres database
I20260924 17:44:40.822862 1058 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-admin
2026-09-24 17:44:41,105 [I] DB FLAVOR :POSTGRES
2026-09-24 17:44:41,105 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:44:41,106 [I] Checking connection..
2026-09-24 17:44:41,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 17:44:41,382 [I] Checking connection passed.
2026-09-24 17:44:41,382 [I] --------- Verifying version history table ---------
2026-09-24 17:44:41,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:44:41,625 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:44:41,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:44:41,873 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:44:41,873 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 17:44:41,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 17:44:42.126 UTC [11938] WARNING: there is no transaction in progress
2026-09-24 17:44:42,132 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 17:44:42,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:44:42,387 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 17:44:42,387 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:44:42,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 17:44:42,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 17:44:42,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r7hf.c.gcp-upstream.internal','N') ;"
2026-09-24 17:44:43,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:44:43,401 [I] Table x_portal_user does not exist in database ranger
2026-09-24 17:44:43,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:44:43,646 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 17:44:43,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:44:43,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:44:44,138 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 17:44:44,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 17:44:45.175 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.194 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.213 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.230 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.248 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.269 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.327 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.335 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.351 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.363 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.374 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.388 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.398 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.405 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.413 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:45.420 UTC [12046] WARNING: there is no transaction in progress
2026-09-24 17:44:46,143 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 17:44:46,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 17:44:46,395 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 17:44:46,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:44:46,639 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 17:44:46,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:44:46,908 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 17:44:46,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:44:47,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:44:47,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r7hf.c.gcp-upstream.internal';"
2026-09-24 17:44:47,629 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 17:44:47,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 17:44:47,890 [I] DB_PATCHES have already been applied
I20260924 17:44:47.895756 1058 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkSMFuh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 17:44:47.895826 1058 mini_ranger.cc:193] Using host: 127.1.8.148
I20260924 17:44:47.899025 1058 mini_ranger.cc:227] Ranger admin URL: http://127.1.8.148:32939
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:44:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 17:44:49.033003 1058 mini_ranger.cc:160] Time spent starting Ranger: real 8.779s user 0.000s sys 0.010s
Sep 24, 2026 5:45:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:45:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 5:45:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:45:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:45:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 17:45:11.597172 1058 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/rangerkms_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/rangerkms_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.8.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.8.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 17:45:11 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271911, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 17:45:11.700968 1058 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 17:45:11.712229 1058 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 17:45:12.648135 1058 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 17:45:12.648224 1058 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms
I20260924 17:45:12.648243 1058 mini_ranger_kms.cc:227] Using postgres at 127.1.8.148:41593
2026-09-24 17:45:12,834 [I] DB FLAVOR :POSTGRES
2026-09-24 17:45:12,835 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:45:12,835 [I] Checking connection
2026-09-24 17:45:12,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 17:45:13,102 [I] connection success
2026-09-24 17:45:13,102 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 17:45:13,102 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 17:45:13,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 17:45:13,386 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 17:45:13,386 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:45:13,386 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 17:45:13,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:41593/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 17:45:13,658 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 17:45:14.119056 1058 mini_ranger_kms.cc:310] Created kms service
I20260924 17:45:14.341256 1058 mini_ranger_kms.cc:326] Created kudu user
I20260924 17:45:14.388358 1058 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 17:45:14 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271911, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:45:14.852818 1058 mini_ranger_kms.cc:384] Added ranger policy
I20260924 17:45:14.852937 1058 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 17:45:14.852962 1058 mini_ranger_kms.cc:240] Using host: 127.1.8.148
I20260924 17:45:14.856603 1058 mini_ranger_kms.cc:276] Ranger KMS PID: 12227
I20260924 17:45:14.856729 1058 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.8.148:34201
Sep 24, 2026 5:45:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:45:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:45:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:45:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:45:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:45:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 17:45:17 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271917, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.8.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:45:17.183295 1058 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.482s user 0.005s sys 0.163s
I20260924 17:45:17.183432 1058 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 17:45:17.183521 1058 mini_ranger_kms.cc:401] 127.1.8.148:34201/kms/v1/keys
Sep 24 17:45:17 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271917, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.8.148@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
Sep 24 17:45:17 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271917, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.8.148@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
Sep 24 17:45:20 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:45:20.306489 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40497
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:40497
--ranger_config_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:34201/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:45:20.390911 12277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:45:20.391163 12277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:45:20.391203 12277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:45:20.392418 12277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:45:20.392470 12277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:45:20.392485 12277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:45:20.392503 12277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:45:20.392516 12277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:45:20.394157 12277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.8.148:34201/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:40497
--ranger_config_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:45:20.394447 12277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:45:20.394718 12277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:45:20.400295 12282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:20.400425 12285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:20.400354 12283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:45:20.400712 12277 server_base.cc:1034] running on GCE node
I20260924 17:45:20.401033 12277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:45:20.401410 12277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:45:20.402603 12277 hybrid_clock.cc:648] HybridClock initialized: now 1790271920402586 us; error 34 us; skew 500 ppm
Sep 24 17:45:20 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:45:20.405707 12277 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:45:20.406241 12277 webserver.cc:460] Webserver started at http://127.1.8.190:39815/ using document root <none> and password file <none>
I20260924 17:45:20.406550 12277 fs_manager.cc:359] Metadata directory not provided
I20260924 17:45:20.406612 12277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:45:20.406756 12277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:45:20 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271920, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:45:20.468636 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "50ef3a877ef146d78fc5b0fab0245198"
format_stamp: "Formatted at 2026-09-24 17:45:20 on dist-test-slave-r7hf"
server_key: "a753b2c06d5a1a5641d4e617798bceac"
server_key_iv: "3557c836d76b70c8496f624df774c237"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:20.469028 12277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "50ef3a877ef146d78fc5b0fab0245198"
format_stamp: "Formatted at 2026-09-24 17:45:20 on dist-test-slave-r7hf"
server_key: "a753b2c06d5a1a5641d4e617798bceac"
server_key_iv: "3557c836d76b70c8496f624df774c237"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:20.470515 12277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:45:20.486865 12292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:20.487061 12277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:45:20.487166 12277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "50ef3a877ef146d78fc5b0fab0245198"
format_stamp: "Formatted at 2026-09-24 17:45:20 on dist-test-slave-r7hf"
server_key: "a753b2c06d5a1a5641d4e617798bceac"
server_key_iv: "3557c836d76b70c8496f624df774c237"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:20.487277 12277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:45:20.499783 12277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:45:20.501969 12277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:45:20.502126 12277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:45:20.507589 12277 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:40497
I20260924 17:45:20.507633 12344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:40497 every 8 connection(s)
I20260924 17:45:20.508025 12277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:45:20.508704 12345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:45:20.511083 12345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198: Bootstrap starting.
I20260924 17:45:20.511688 12345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198: Neither blocks nor log segments found. Creating new log.
I20260924 17:45:20.511970 12345 log.cc:824] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198: Log is configured to *not* fsync() on all Append() calls
I20260924 17:45:20.512770 12345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198: No bootstrap required, opened a new log
I20260924 17:45:20.513116 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 12277
I20260924 17:45:20.513243 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:45:20.513455 1058 external_mini_cluster.cc:1428] Setting key 8d7998ea4770307c6bfecc3d53a1e486
I20260924 17:45:20.515408 12345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef3a877ef146d78fc5b0fab0245198" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40497 } }
I20260924 17:45:20.515581 12345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:45:20.515645 12345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ef3a877ef146d78fc5b0fab0245198, State: Initialized, Role: FOLLOWER
I20260924 17:45:20.515820 12345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef3a877ef146d78fc5b0fab0245198" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40497 } }
I20260924 17:45:20.515916 12345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:45:20.515950 12345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:45:20.516004 12345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:45:20.516782 12345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef3a877ef146d78fc5b0fab0245198" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40497 } }
I20260924 17:45:20.516904 12345 leader_election.cc:302] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ef3a877ef146d78fc5b0fab0245198; no voters:
I20260924 17:45:20.517108 12345 leader_election.cc:288] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:45:20.517139 12350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:45:20.517350 12350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [term 1 LEADER]: Becoming Leader. State: Replica: 50ef3a877ef146d78fc5b0fab0245198, State: Running, Role: LEADER
I20260924 17:45:20.517462 12345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:45:20.517473 12350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef3a877ef146d78fc5b0fab0245198" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40497 } }
I20260924 17:45:20.518085 12352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ef3a877ef146d78fc5b0fab0245198. Latest consensus state: current_term: 1 leader_uuid: "50ef3a877ef146d78fc5b0fab0245198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef3a877ef146d78fc5b0fab0245198" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40497 } } }
I20260924 17:45:20.518244 12352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [sys.catalog]: This master's current role is: LEADER
I20260924 17:45:20.518543 12359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:45:20.519095 12359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:45:20.517958 12350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ef3a877ef146d78fc5b0fab0245198" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef3a877ef146d78fc5b0fab0245198" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40497 } } }
I20260924 17:45:20.519200 12350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198 [sys.catalog]: This master's current role is: LEADER
I20260924 17:45:20.520269 12345 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 17:45:20 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:45:20.530381 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:20.514054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56200 (local address 127.1.8.190:40497)
0924 17:45:20.514253 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:20.514263 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:20.514295 (+ 32us) server_negotiation.cc:407] Connection header received
0924 17:45:20.515408 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:20.515419 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:20.515494 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:20.515663 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:45:20.516559 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:20.517927 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:20.519124 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:20.519766 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:20.519880 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:20.525160 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:45:20.525202 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:45:20.525202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:45:20.525289 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:45:20.528030 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:20.528522 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:20.528526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:20.528526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:20.528631 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:20.528865 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:20.528865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:20.528866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:20.529268 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:45:20.529891 (+ 623us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:20.529966 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 17:45:20.530151 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260924 17:45:20.534590 12359 catalog_manager.cc:1347] Generated new cluster ID: e82e136881c24b43ab2af29d3aa02a08
I20260924 17:45:20.534652 12359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:45:20.549110 12359 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:45:20.549989 12359 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:45:20.559654 12359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198: Generated new TSK 0
I20260924 17:45:20.560012 12359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:45:21.327065 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:34201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40497
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:45:21.454066 12390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:45:21.454427 12390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:45:21.454473 12390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:45:21.456336 12390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:45:21.456414 12390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:45:21.456459 12390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:45:21.458812 12390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.8.148:34201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:45:21.459110 12390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:45:21.459468 12390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:45:21.465070 12396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:21.465162 12398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:21.465348 12395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:45:21.465605 12390 server_base.cc:1034] running on GCE node
I20260924 17:45:21.465821 12390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:45:21.466092 12390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:45:21.467273 12390 hybrid_clock.cc:648] HybridClock initialized: now 1790271921467257 us; error 27 us; skew 500 ppm
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:45:21.470278 12390 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:45:21.470768 12390 webserver.cc:460] Webserver started at http://127.1.8.129:45547/ using document root <none> and password file <none>
I20260924 17:45:21.471019 12390 fs_manager.cc:359] Metadata directory not provided
I20260924 17:45:21.471088 12390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:45:21.471254 12390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:45:21.485664 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "866c0b139f7e402682aec2dc428ca785"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "08a4bfd6343f14e7d8689f1e3bec01da"
server_key_iv: "3fa984ad1a69afa24075008fecbcfe55"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:21.486142 12390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "866c0b139f7e402682aec2dc428ca785"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "08a4bfd6343f14e7d8689f1e3bec01da"
server_key_iv: "3fa984ad1a69afa24075008fecbcfe55"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:21.487922 12390 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 17:45:21.504982 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:21.505244 12390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:45:21.505362 12390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "866c0b139f7e402682aec2dc428ca785"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "08a4bfd6343f14e7d8689f1e3bec01da"
server_key_iv: "3fa984ad1a69afa24075008fecbcfe55"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:21.505465 12390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:45:21.519675 12390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:45:21.521817 12390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:45:21.521986 12390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:45:21.522249 12390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:45:21.522673 12390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:45:21.522727 12390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:21.522799 12390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:45:21.522843 12390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:21.530180 12390 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:38599
I20260924 17:45:21.530249 12520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:38599 every 8 connection(s)
I20260924 17:45:21.530660 12390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:45:21.534585 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 12390
I20260924 17:45:21.534689 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:45:21.534886 1058 external_mini_cluster.cc:1428] Setting key 228e95fc1e153ecdf242b53411c62bf0
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:45:21.541468 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:21.531577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:34971 (local address 127.1.8.190:40497)
0924 17:45:21.531717 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:21.531719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:21.531924 (+ 205us) server_negotiation.cc:407] Connection header received
0924 17:45:21.532463 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:21.532465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:21.532479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:21.532553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:45:21.533146 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.533671 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.534900 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.535263 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.535324 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:21.537843 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:45:21.537845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:45:21.537846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:45:21.537854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:45:21.540033 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:21.540451 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:21.540452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:21.540453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:21.540521 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:21.540778 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:21.540779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:21.540779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:21.540900 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:45:21.540967 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:21.541280 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 17:45:21.541367 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260924 17:45:21.541656 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:21.531573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40497 (local address 127.1.8.129:34971)
0924 17:45:21.531784 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:21.531825 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:21.532357 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:21.532636 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:21.532640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:21.532831 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:45:21.533046 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:21.533051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.533770 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.533771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:21.534795 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:21.534798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.535390 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.535391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:21.535541 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:21.536199 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:45:21.536211 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:45:21.537735 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:45:21.540123 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:45:21.540130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:45:21.540133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:45:21.540367 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:45:21.540588 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:45:21.540589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:45:21.540589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:45:21.540702 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:45:21.540970 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:45:21.540977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:45:21.541139 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 17:45:21.541250 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:45:21.541406 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260924 17:45:21.542327 12521 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40497
I20260924 17:45:21.542464 12521 heartbeater.cc:464] Registering TS with master...
I20260924 17:45:21.542655 12521 heartbeater.cc:511] Master 127.1.8.190:40497 requested a full tablet report, sending...
I20260924 17:45:21.543227 12309 ts_manager.cc:194] Registered new tserver with Master: 866c0b139f7e402682aec2dc428ca785 (127.1.8.129:38599)
I20260924 17:45:21.543845 12309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:34971
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:45:21.573732 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:34201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40497
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:45:21.681130 12527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:45:21.681382 12527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:45:21.681425 12527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:45:21.682857 12527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:45:21.682924 12527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:45:21.682972 12527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:45:21.684644 12527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.8.148:34201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:45:21.684933 12527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:45:21.685210 12527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 17:45:21.690734 12537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:21.690732 12534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:21.691440 12535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:45:21.691993 12527 server_base.cc:1034] running on GCE node
I20260924 17:45:21.692235 12527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:45:21.692504 12527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:45:21.699916 12527 hybrid_clock.cc:648] HybridClock initialized: now 1790271921699874 us; error 44 us; skew 500 ppm
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:45:21.703845 12527 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:45:21.704432 12527 webserver.cc:460] Webserver started at http://127.1.8.130:42239/ using document root <none> and password file <none>
I20260924 17:45:21.704734 12527 fs_manager.cc:359] Metadata directory not provided
I20260924 17:45:21.704793 12527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:45:21.704927 12527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:45:21.721874 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "0004a54ba72c474a8fcece1eabe2fbff"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "a7e2848bf74b29c49e843b2d73dbfae5"
server_key_iv: "0eedb287ee328a481c0b21990feb9e1a"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:21.722532 12527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "0004a54ba72c474a8fcece1eabe2fbff"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "a7e2848bf74b29c49e843b2d73dbfae5"
server_key_iv: "0eedb287ee328a481c0b21990feb9e1a"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:21.733891 12527 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260924 17:45:21.748049 12544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:21.748317 12527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:45:21.748400 12527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "0004a54ba72c474a8fcece1eabe2fbff"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "a7e2848bf74b29c49e843b2d73dbfae5"
server_key_iv: "0eedb287ee328a481c0b21990feb9e1a"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:21.748492 12527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:45:21.763942 12527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:45:21.766571 12527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:45:21.766847 12527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:45:21.767204 12527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:45:21.767606 12527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:45:21.767696 12527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:21.767771 12527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:45:21.767828 12527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:21.777433 12527 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:37933
I20260924 17:45:21.778055 12527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:45:21.778717 12658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:37933 every 8 connection(s)
I20260924 17:45:21.784210 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 12527
I20260924 17:45:21.784329 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:45:21.784518 1058 external_mini_cluster.cc:1428] Setting key 8dc8aea1dd6103eeb4ae110759f1d0cf
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:45:21.791225 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:21.779697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:56101 (local address 127.1.8.190:40497)
0924 17:45:21.779750 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:21.779752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:21.780045 (+ 293us) server_negotiation.cc:407] Connection header received
0924 17:45:21.780771 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:21.780773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:21.780790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:21.780827 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:45:21.781846 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.782684 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.784237 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.784556 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.784606 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:21.787440 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:45:21.787442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:45:21.787442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:45:21.787451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:45:21.789254 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:21.789843 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:21.789844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:21.789844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:21.789910 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:21.790306 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:21.790307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:21.790307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:21.790493 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:45:21.790539 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:21.791020 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 17:45:21.791122 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:45:21.792112 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:21.779682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40497 (local address 127.1.8.130:56101)
0924 17:45:21.779885 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:21.779940 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:21.780632 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:21.781068 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:21.781073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:21.781329 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:45:21.781728 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:21.781735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.782806 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.782807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:21.784126 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:21.784128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:21.784709 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:21.784710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:21.784855 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:21.785505 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:45:21.785516 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:45:21.787315 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:45:21.789363 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:45:21.789370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:45:21.789374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:45:21.789707 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:45:21.790019 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:45:21.790019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:45:21.790020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:45:21.790221 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:45:21.790591 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:45:21.790598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:45:21.790795 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 17:45:21.791652 (+ 857us) client_negotiation.cc:240] Negotiation successful
0924 17:45:21.791839 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
I20260924 17:45:21.792891 12659 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40497
I20260924 17:45:21.793004 12659 heartbeater.cc:464] Registering TS with master...
I20260924 17:45:21.793208 12659 heartbeater.cc:511] Master 127.1.8.190:40497 requested a full tablet report, sending...
I20260924 17:45:21.793673 12309 ts_manager.cc:194] Registered new tserver with Master: 0004a54ba72c474a8fcece1eabe2fbff (127.1.8.130:37933)
I20260924 17:45:21.794176 12309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:56101
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:45:21.833647 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:34201/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40497
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:45:21.965067 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:45:21.965453 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:45:21.965543 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:45:21.967751 12664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:45:21.968017 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:45:21.968117 12664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:45:21.970496 12664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.8.148:34201/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:45:21.971025 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:45:21.971359 12664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:45:21.978740 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:21.979446 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:45:21.979666 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:45:21.980224 12664 server_base.cc:1034] running on GCE node
I20260924 17:45:21.980469 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:45:21.980773 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:45:21.981956 12664 hybrid_clock.cc:648] HybridClock initialized: now 1790271921981926 us; error 42 us; skew 500 ppm
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:45:21.985008 12664 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:45:21.985596 12664 webserver.cc:460] Webserver started at http://127.1.8.131:42125/ using document root <none> and password file <none>
I20260924 17:45:21.985844 12664 fs_manager.cc:359] Metadata directory not provided
I20260924 17:45:21.985915 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:45:21.986064 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:45:21 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:45:21.999739 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "8a165c39addd451f9b2a54e46907c354"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "fa6c57e15805235fe4fdf0deaf1f88a3"
server_key_iv: "0f5cdc5ddb9067e339650ff07123b80b"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:22.000213 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "8a165c39addd451f9b2a54e46907c354"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "fa6c57e15805235fe4fdf0deaf1f88a3"
server_key_iv: "0f5cdc5ddb9067e339650ff07123b80b"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:22.001708 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:45:22.028162 12679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:22.028431 12664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 17:45:22.028530 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "8a165c39addd451f9b2a54e46907c354"
format_stamp: "Formatted at 2026-09-24 17:45:21 on dist-test-slave-r7hf"
server_key: "fa6c57e15805235fe4fdf0deaf1f88a3"
server_key_iv: "0f5cdc5ddb9067e339650ff07123b80b"
server_key_version: "kuduclusterkey@0"
I20260924 17:45:22.028640 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:45:22.052512 12664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:45:22.059998 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:45:22.060247 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:45:22.060921 12664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:45:22.061390 12664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:45:22.061482 12664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:22.061564 12664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:45:22.061628 12664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:45:22.069439 12664 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:38875
I20260924 17:45:22.070683 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:45:22.075888 12792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:38875 every 8 connection(s)
I20260924 17:45:22.080276 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 12664
I20260924 17:45:22.080404 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:45:22.080610 1058 external_mini_cluster.cc:1428] Setting key d0467dcb722f0975ced7daf48535a289
Sep 24 17:45:22 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:45:22.090126 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.077050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:32901 (local address 127.1.8.190:40497)
0924 17:45:22.077095 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.077097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.077463 (+ 366us) server_negotiation.cc:407] Connection header received
0924 17:45:22.078205 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.078207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.078224 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.078259 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:45:22.079214 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.080013 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.083166 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.083451 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.083501 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.086483 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:45:22.086484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:45:22.086485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:45:22.086494 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:45:22.088335 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:22.088882 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:22.088883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:22.088884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:22.088944 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:22.089235 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:22.089235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:22.089236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:22.089392 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:45:22.089448 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.089917 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.090019 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:45:22.091010 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.076724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40497 (local address 127.1.8.131:32901)
0924 17:45:22.077320 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.077371 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:22.078055 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:22.078523 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:22.078527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:22.078794 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:45:22.079097 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.079104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.081646 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.081647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.083053 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.083057 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.083607 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.083608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.083762 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.084402 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:45:22.084413 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:45:22.086200 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:45:22.088443 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:45:22.088450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:45:22.088453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:45:22.088786 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:45:22.089013 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:45:22.089014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:45:22.089014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:45:22.089156 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:45:22.089484 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:45:22.089491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:45:22.089743 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 17:45:22.090567 (+ 824us) client_negotiation.cc:240] Negotiation successful
0924 17:45:22.090759 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":108,"threads_started":1}
I20260924 17:45:22.091786 12793 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40497
I20260924 17:45:22.091883 12793 heartbeater.cc:464] Registering TS with master...
I20260924 17:45:22.092083 12793 heartbeater.cc:511] Master 127.1.8.190:40497 requested a full tablet report, sending...
I20260924 17:45:22.092952 12309 ts_manager.cc:194] Registered new tserver with Master: 8a165c39addd451f9b2a54e46907c354 (127.1.8.131:38875)
I20260924 17:45:22.093467 12309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:32901
I20260924 17:45:22.104542 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:45:22.114921 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.106489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.1.8.190:40497)
0924 17:45:22.106533 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.106536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.106820 (+ 284us) server_negotiation.cc:407] Connection header received
0924 17:45:22.106925 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.106926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.106937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.106967 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:45:22.107770 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.108587 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.109743 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.110041 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.110087 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.111826 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:45:22.111828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:45:22.111828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:45:22.111837 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:45:22.113576 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:22.113996 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:22.113997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:22.113997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:22.114055 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:45:22.114253 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:45:22.114254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:45:22.114254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:45:22.114426 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:45:22.114505 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.114730 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.114819 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:45:22.117386 12309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:45:22.119957 12309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:45:22.138726 12810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.123858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:38599 (local address 127.0.0.1:53020)
0924 17:45:22.125862 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.125873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:22.125925 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:22.126296 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:22.126299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:22.126301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:45:22.126369 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.126371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.130170 (+ 3799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.130171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.131428 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.131431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.138443 (+ 7012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.138444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.138504 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.138507 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:45:22.138567 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:45:22.138634 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1985,"thread_start_us":70,"threads_started":1}
I20260924 17:45:22.141084 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.123665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:37933 (local address 127.0.0.1:47594)
0924 17:45:22.125252 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.125307 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:22.125371 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:22.126446 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:22.126447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:22.126448 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:45:22.126478 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.126479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.128684 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.128685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.137132 (+ 8447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.137135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.138233 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.138234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.138346 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.138364 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 17:45:22.140942 (+ 2578us) client_negotiation.cc:240] Negotiation successful
0924 17:45:22.140993 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1557,"thread_start_us":62,"threads_started":1}
I20260924 17:45:22.141875 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.124650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:38875 (local address 127.0.0.1:46790)
0924 17:45:22.131567 (+ 6917us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.131579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:22.131625 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:22.133000 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:22.133001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:22.133003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:45:22.133055 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.133084 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.134450 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.134451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.139491 (+ 5040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.139494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.141639 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.141640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.141707 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.141712 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:45:22.141760 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 17:45:22.141801 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6896,"thread_start_us":76,"threads_started":1}
I20260924 17:45:22.142262 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.124350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.1.8.129:38599)
0924 17:45:22.125805 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.125809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.126077 (+ 268us) server_negotiation.cc:407] Connection header received
0924 17:45:22.126113 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.126115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.126131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.126200 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:45:22.127119 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.130066 (+ 2947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.136032 (+ 5966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.136855 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.136935 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.137003 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.142124 (+ 5121us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.142187 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":53,"threads_started":1}
I20260924 17:45:22.142661 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.124149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.1.8.130:37933)
0924 17:45:22.125516 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.125522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.125532 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:45:22.125574 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.125576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.125594 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.125672 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:45:22.126572 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.128584 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.137261 (+ 8677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.138040 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.138113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.138193 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.142524 (+ 4331us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.142579 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":93,"threads_started":1}
I20260924 17:45:22.143018 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.124901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.1.8.131:38875)
0924 17:45:22.126053 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.126057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.131689 (+ 5632us) server_negotiation.cc:407] Connection header received
0924 17:45:22.131739 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.131742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.131760 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.131838 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:45:22.133174 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.134337 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.139612 (+ 5275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.140733 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.140806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.140877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.142867 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.142915 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":72,"threads_started":1}
I20260924 17:45:22.143980 12455 tablet_service.cc:1467] Processing CreateTablet for tablet b32e2dc9b7cf44f38e9f5e5e064224ca (DEFAULT_TABLE table=test-table [id=1ae41dbbb17a48eea292afd19738f8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:45:22.144377 12455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32e2dc9b7cf44f38e9f5e5e064224ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:45:22.149526 12593 tablet_service.cc:1467] Processing CreateTablet for tablet b32e2dc9b7cf44f38e9f5e5e064224ca (DEFAULT_TABLE table=test-table [id=1ae41dbbb17a48eea292afd19738f8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:45:22.149909 12593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32e2dc9b7cf44f38e9f5e5e064224ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:45:22.154943 12727 tablet_service.cc:1467] Processing CreateTablet for tablet b32e2dc9b7cf44f38e9f5e5e064224ca (DEFAULT_TABLE table=test-table [id=1ae41dbbb17a48eea292afd19738f8b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:45:22.155413 12727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b32e2dc9b7cf44f38e9f5e5e064224ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:45:22.160290 12813 tablet_bootstrap.cc:492] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Bootstrap starting.
I20260924 17:45:22.161396 12812 tablet_bootstrap.cc:492] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: Bootstrap starting.
I20260924 17:45:22.162451 12813 tablet_bootstrap.cc:654] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Neither blocks nor log segments found. Creating new log.
I20260924 17:45:22.162731 12814 tablet_bootstrap.cc:492] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Bootstrap starting.
I20260924 17:45:22.163682 12812 tablet_bootstrap.cc:654] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: Neither blocks nor log segments found. Creating new log.
I20260924 17:45:22.164837 12814 tablet_bootstrap.cc:654] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Neither blocks nor log segments found. Creating new log.
I20260924 17:45:22.165297 12813 log.cc:824] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Log is configured to *not* fsync() on all Append() calls
I20260924 17:45:22.165753 12812 log.cc:824] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: Log is configured to *not* fsync() on all Append() calls
I20260924 17:45:22.166206 12814 log.cc:824] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Log is configured to *not* fsync() on all Append() calls
I20260924 17:45:22.167919 12813 tablet_bootstrap.cc:492] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: No bootstrap required, opened a new log
I20260924 17:45:22.168064 12813 ts_tablet_manager.cc:1397] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260924 17:45:22.171615 12813 raft_consensus.cc:348] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.171996 12813 raft_consensus.cc:374] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:45:22.172096 12813 raft_consensus.cc:729] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0004a54ba72c474a8fcece1eabe2fbff, State: Initialized, Role: FOLLOWER
I20260924 17:45:22.172325 12813 consensus_queue.cc:260] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.172778 12813 ts_tablet_manager.cc:1428] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260924 17:45:22.172981 12659 heartbeater.cc:503] Master 127.1.8.190:40497 was elected leader, sending a full tablet report...
I20260924 17:45:22.174786 12812 tablet_bootstrap.cc:492] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: No bootstrap required, opened a new log
I20260924 17:45:22.174944 12812 ts_tablet_manager.cc:1397] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20260924 17:45:22.178191 12812 raft_consensus.cc:348] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.178529 12812 raft_consensus.cc:374] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:45:22.178623 12812 raft_consensus.cc:729] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 866c0b139f7e402682aec2dc428ca785, State: Initialized, Role: FOLLOWER
I20260924 17:45:22.178848 12812 consensus_queue.cc:260] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.179243 12812 ts_tablet_manager.cc:1428] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260924 17:45:22.179445 12521 heartbeater.cc:503] Master 127.1.8.190:40497 was elected leader, sending a full tablet report...
I20260924 17:45:22.180270 12814 tablet_bootstrap.cc:492] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: No bootstrap required, opened a new log
I20260924 17:45:22.180431 12814 ts_tablet_manager.cc:1397] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20260924 17:45:22.182788 12814 raft_consensus.cc:348] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.183638 12814 raft_consensus.cc:374] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:45:22.183743 12814 raft_consensus.cc:729] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a165c39addd451f9b2a54e46907c354, State: Initialized, Role: FOLLOWER
I20260924 17:45:22.183985 12814 consensus_queue.cc:260] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.184532 12814 ts_tablet_manager.cc:1428] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 17:45:22.184719 12793 heartbeater.cc:503] Master 127.1.8.190:40497 was elected leader, sending a full tablet report...
Sep 24 17:45:22 dist-test-slave-r7hf krb5kdc[11833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271921, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
W20260924 17:45:22.281889 12522 tablet.cc:2369] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:45:22.282301 12660 tablet.cc:2369] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:45:22.327605 12794 tablet.cc:2369] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:45:22.338958 12820 raft_consensus.cc:482] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:45:22.339130 12820 raft_consensus.cc:504] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.339560 12820 leader_election.cc:288] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0004a54ba72c474a8fcece1eabe2fbff (127.1.8.130:37933), 866c0b139f7e402682aec2dc428ca785 (127.1.8.129:38599)
I20260924 17:45:22.344039 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.339964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:37933 (local address 127.1.8.131:56659)
0924 17:45:22.340264 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.340276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:22.340323 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:22.340762 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:22.340763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:22.340765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:45:22.340835 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.340838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.341786 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.341787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.343020 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.343023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.343783 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.343784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.343861 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.343865 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:45:22.343915 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:45:22.343956 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260924 17:45:22.344820 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.340464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:56659 (local address 127.1.8.130:37933)
0924 17:45:22.340509 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.340511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.340519 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:45:22.340547 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.340548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.340558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.340605 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:45:22.340922 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.341684 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.343123 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.343678 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.343740 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.343751 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.344700 (+ 949us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.344750 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:45:22.345115 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e2dc9b7cf44f38e9f5e5e064224ca" candidate_uuid: "8a165c39addd451f9b2a54e46907c354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0004a54ba72c474a8fcece1eabe2fbff" is_pre_election: true
I20260924 17:45:22.345305 12611 raft_consensus.cc:2454] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a165c39addd451f9b2a54e46907c354 in term 0.
I20260924 17:45:22.346309 12681 leader_election.cc:302] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0004a54ba72c474a8fcece1eabe2fbff, 8a165c39addd451f9b2a54e46907c354; no voters:
I20260924 17:45:22.346549 12820 raft_consensus.cc:2781] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:45:22.346601 12820 raft_consensus.cc:482] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:45:22.346622 12820 raft_consensus.cc:3037] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:45:22.347529 12820 raft_consensus.cc:504] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.347666 12820 leader_election.cc:288] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [CANDIDATE]: Term 1 election: Requested vote from peers 0004a54ba72c474a8fcece1eabe2fbff (127.1.8.130:37933), 866c0b139f7e402682aec2dc428ca785 (127.1.8.129:38599)
I20260924 17:45:22.347963 12611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e2dc9b7cf44f38e9f5e5e064224ca" candidate_uuid: "8a165c39addd451f9b2a54e46907c354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0004a54ba72c474a8fcece1eabe2fbff"
I20260924 17:45:22.348044 12611 raft_consensus.cc:3037] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:45:22.348876 12611 raft_consensus.cc:2454] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a165c39addd451f9b2a54e46907c354 in term 1.
I20260924 17:45:22.349105 12681 leader_election.cc:302] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0004a54ba72c474a8fcece1eabe2fbff, 8a165c39addd451f9b2a54e46907c354; no voters:
I20260924 17:45:22.349217 12820 raft_consensus.cc:2781] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:45:22.349388 12820 raft_consensus.cc:686] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 1 LEADER]: Becoming Leader. State: Replica: 8a165c39addd451f9b2a54e46907c354, State: Running, Role: LEADER
I20260924 17:45:22.349475 12820 consensus_queue.cc:237] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } }
I20260924 17:45:22.350255 12309 catalog_manager.cc:5495] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a165c39addd451f9b2a54e46907c354 (127.1.8.131). New cstate: current_term: 1 leader_uuid: "8a165c39addd451f9b2a54e46907c354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a165c39addd451f9b2a54e46907c354" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38875 } health_report { overall_health: HEALTHY } } }
I20260924 17:45:22.353698 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.340066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:38599 (local address 127.1.8.131:47789)
0924 17:45:22.344131 (+ 4065us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.344145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:45:22.344188 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:45:22.345151 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:45:22.345152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:45:22.345154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:45:22.345188 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.345189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.350951 (+ 5762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.350952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.352334 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:45:22.352346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.353457 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.353459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:45:22.353524 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.353528 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:45:22.353579 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:45:22.353621 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4052,"thread_start_us":104,"threads_started":1}
I20260924 17:45:22.354024 12809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.340688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:47789 (local address 127.1.8.129:38599)
0924 17:45:22.340729 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.340731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.344244 (+ 3513us) server_negotiation.cc:407] Connection header received
0924 17:45:22.344274 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.344275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.344289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.344333 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:45:22.345276 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.346005 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.352707 (+ 6702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.353290 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.353391 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.353404 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.353903 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.353958 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:45:22.354588 12475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e2dc9b7cf44f38e9f5e5e064224ca" candidate_uuid: "8a165c39addd451f9b2a54e46907c354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866c0b139f7e402682aec2dc428ca785" is_pre_election: true
I20260924 17:45:22.354771 12475 raft_consensus.cc:2454] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a165c39addd451f9b2a54e46907c354 in term 0.
I20260924 17:45:22.354902 12474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e2dc9b7cf44f38e9f5e5e064224ca" candidate_uuid: "8a165c39addd451f9b2a54e46907c354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "866c0b139f7e402682aec2dc428ca785"
I20260924 17:45:22.354959 12474 raft_consensus.cc:3037] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:45:22.355871 12474 raft_consensus.cc:2454] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a165c39addd451f9b2a54e46907c354 in term 1.
I20260924 17:45:22.413829 12811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.411269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.8.131:38875)
0924 17:45:22.411331 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:45:22.411333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:45:22.411364 (+ 31us) server_negotiation.cc:407] Connection header received
0924 17:45:22.411463 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:45:22.411464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:45:22.411487 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:45:22.411574 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:45:22.411692 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.412171 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.412961 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:45:22.413190 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:45:22.413266 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:45:22.413447 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:45:22.413568 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:45:22.413666 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 17:45:22.413686 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 17:45:22.413708 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:45:22.415848 12611 raft_consensus.cc:1261] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 1 FOLLOWER]: Refusing update from remote peer 8a165c39addd451f9b2a54e46907c354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:45:22.415836 12474 raft_consensus.cc:1261] T b32e2dc9b7cf44f38e9f5e5e064224ca P 866c0b139f7e402682aec2dc428ca785 [term 1 FOLLOWER]: Refusing update from remote peer 8a165c39addd451f9b2a54e46907c354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:45:22.416231 12822 consensus_queue.cc:1035] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "866c0b139f7e402682aec2dc428ca785" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:45:22.416360 12820 consensus_queue.cc:1035] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0004a54ba72c474a8fcece1eabe2fbff" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:45:22.420730 12825 mvcc.cc:205] Tried to move back new op lower bound from 7332953794212405248 to 7332953793943883776. Current Snapshot: MvccSnapshot[applied={T|T < 7332953794212405248}]
I20260924 17:45:22.420858 12827 mvcc.cc:205] Tried to move back new op lower bound from 7332953794212405248 to 7332953793943883776. Current Snapshot: MvccSnapshot[applied={T|T < 7332953794212405248}]
I20260924 17:45:22.422158 12826 mvcc.cc:205] Tried to move back new op lower bound from 7332953794212405248 to 7332953793943883776. Current Snapshot: MvccSnapshot[applied={T|T < 7332953794212405248 or (T in {7332953794212405248})}]
I20260924 17:45:22.424278 12309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:45:22.424382 12309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:45:22.425319 12309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50ef3a877ef146d78fc5b0fab0245198: Sending DeleteTablet for 3 replicas of tablet b32e2dc9b7cf44f38e9f5e5e064224ca
I20260924 17:45:22.425716 12727 tablet_service.cc:1514] Processing DeleteTablet for tablet b32e2dc9b7cf44f38e9f5e5e064224ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:45:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:46790
I20260924 17:45:22.425930 12593 tablet_service.cc:1514] Processing DeleteTablet for tablet b32e2dc9b7cf44f38e9f5e5e064224ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:45:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:47594
I20260924 17:45:22.426154 12455 tablet_service.cc:1514] Processing DeleteTablet for tablet b32e2dc9b7cf44f38e9f5e5e064224ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:45:22 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:53020
I20260924 17:45:22.426918 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 12390
I20260924 17:45:22.434518 12835 tablet_replica.cc:330] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: stopping tablet replica
I20260924 17:45:22.434576 12834 tablet_replica.cc:330] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: stopping tablet replica
I20260924 17:45:22.434751 12835 raft_consensus.cc:2229] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:45:22.434882 12834 raft_consensus.cc:2229] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:45:22.434966 12835 raft_consensus.cc:2258] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:45:22.435104 12834 raft_consensus.cc:2258] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:45:22.435631 12835 ts_tablet_manager.cc:1905] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:45:22.435763 12834 ts_tablet_manager.cc:1905] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:45:22.437770 12834 ts_tablet_manager.cc:1918] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:45:22.437954 12834 log.cc:1196] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/b32e2dc9b7cf44f38e9f5e5e064224ca
I20260924 17:45:22.438336 12834 ts_tablet_manager.cc:1939] T b32e2dc9b7cf44f38e9f5e5e064224ca P 8a165c39addd451f9b2a54e46907c354: Deleting consensus metadata
I20260924 17:45:22.438961 12293 catalog_manager.cc:4841] TS 8a165c39addd451f9b2a54e46907c354 (127.1.8.131:38875): tablet b32e2dc9b7cf44f38e9f5e5e064224ca (table test-table [id=1ae41dbbb17a48eea292afd19738f8b2]) successfully deleted
I20260924 17:45:22.441085 12835 ts_tablet_manager.cc:1918] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:45:22.441171 12835 log.cc:1196] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/b32e2dc9b7cf44f38e9f5e5e064224ca
I20260924 17:45:22.441431 12835 ts_tablet_manager.cc:1939] T b32e2dc9b7cf44f38e9f5e5e064224ca P 0004a54ba72c474a8fcece1eabe2fbff: Deleting consensus metadata
W20260924 17:45:22.441730 12294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:38599 (error 108)
I20260924 17:45:22.442296 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 12527
I20260924 17:45:22.442875 12294 catalog_manager.cc:4841] TS 0004a54ba72c474a8fcece1eabe2fbff (127.1.8.130:37933): tablet b32e2dc9b7cf44f38e9f5e5e064224ca (table test-table [id=1ae41dbbb17a48eea292afd19738f8b2]) successfully deleted
I20260924 17:45:22.443732 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:45:22.443589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:38599 (local address 127.0.0.1:53032)
0924 17:45:22.443646 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 17:45:22.443697 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:38599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 17:45:22.443949 12294 catalog_manager.cc:4584] TS 866c0b139f7e402682aec2dc428ca785 (127.1.8.129:38599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b32e2dc9b7cf44f38e9f5e5e064224ca: Network error: Client connection negotiation failed: client connection to 127.1.8.129:38599: connect: Connection refused (error 111)
I20260924 17:45:22.448585 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 12664
I20260924 17:45:22.454962 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 12277
I20260924 17:45:22.461141 1058 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 17:45:22.806360 1058 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 17:45:22.806555 1058 mini_ranger.cc:66] Stopping Ranger...
I20260924 17:45:23.151757 1058 mini_ranger.cc:68] Stopped Ranger
2026-09-24 17:45:23.151 UTC [11863] LOG: received smart shutdown request
2026-09-24 17:45:23.155 UTC [11863] LOG: background worker "logical replication launcher" (PID 11872) exited with exit code 1
2026-09-24 17:45:23.155 UTC [11867] LOG: shutting down
2026-09-24 17:45:23.356 UTC [11863] LOG: database system is shut down
2026-09-24T17:45:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48232 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:45:23 dist-test-slave-r7hf krb5kdc[12846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:45:23 dist-test-slave-r7hf krb5kdc[12846](info): set up 2 sockets
Sep 24 17:45:23 dist-test-slave-r7hf krb5kdc[12846](info): commencing operation
krb5kdc: starting...
W20260924 17:45:24.518519 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:45:24 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:45:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/rangeradmin_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/rangeradmin_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/rangerlookup_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/rangerlookup_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
I20260924 17:45:24.693840 1058 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskkSMFuh/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/postgres -l logfile start
2026-09-24 17:45:27.360 UTC [12876] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 17:45:27.360 UTC [12876] LOG: listening on IPv4 address "127.1.8.148", port 55045
2026-09-24 17:45:27.364 UTC [12876] LOG: listening on Unix socket "/tmp/.s.PGSQL.55045"
2026-09-24 17:45:27.385 UTC [12879] LOG: database system was shut down at 2026-09-24 17:45:25 UTC
2026-09-24 17:45:27.389 UTC [12876] LOG: database system is ready to accept connections
I20260924 17:45:28.366381 1058 mini_postgres.cc:96] Postgres bound to 55045
2026-09-24 17:45:28.372 UTC [12887] FATAL: database "slave" does not exist
127.1.8.148:55045 - accepting connections
I20260924 17:45:28.373330 1058 mini_ranger.cc:161] Starting Ranger...
I20260924 17:45:28.388839 1058 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 17:45:28.937078 1058 mini_ranger.cc:87] Created ranger Postgres database
I20260924 17:45:28.937166 1058 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-admin
2026-09-24 17:45:29,229 [I] DB FLAVOR :POSTGRES
2026-09-24 17:45:29,229 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:45:29,230 [I] Checking connection..
2026-09-24 17:45:29,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 17:45:29,486 [I] Checking connection passed.
2026-09-24 17:45:29,486 [I] --------- Verifying version history table ---------
2026-09-24 17:45:29,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:45:29,745 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:45:29,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:45:29,983 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 17:45:29,984 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 17:45:29,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 17:45:30.218 UTC [12951] WARNING: there is no transaction in progress
2026-09-24 17:45:30,225 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 17:45:30,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 17:45:30,473 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 17:45:30,473 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:45:30,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 17:45:30,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 17:45:30,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r7hf.c.gcp-upstream.internal','N') ;"
2026-09-24 17:45:31,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:45:31,489 [I] Table x_portal_user does not exist in database ranger
2026-09-24 17:45:31,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:45:31,739 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 17:45:31,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:45:31,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:45:32,239 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 17:45:32,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 17:45:33.252 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.268 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.286 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.302 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.319 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.338 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.403 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.410 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.426 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.437 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.448 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.462 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.472 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.479 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.486 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:33.494 UTC [13059] WARNING: there is no transaction in progress
2026-09-24 17:45:34,201 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 17:45:34,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 17:45:34,448 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 17:45:34,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 17:45:34,686 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 17:45:34,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 17:45:34,933 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 17:45:34,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 17:45:35,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 17:45:35,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r7hf.c.gcp-upstream.internal';"
2026-09-24 17:45:35,731 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 17:45:35,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 17:45:35,969 [I] DB_PATCHES have already been applied
I20260924 17:45:35.975039 1058 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-admin:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskkSMFuh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 17:45:35.975104 1058 mini_ranger.cc:193] Using host: 127.1.8.148
I20260924 17:45:35.978255 1058 mini_ranger.cc:227] Ranger admin URL: http://127.1.8.148:57059
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:45:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 17:45:37.100069 1058 mini_ranger.cc:160] Time spent starting Ranger: real 8.727s user 0.005s sys 0.004s
Sep 24, 2026 5:45:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:45:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 5:45:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:45:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:45:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 17:46:00.345394 1058 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.8.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.8.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/rangerkms_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.8.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/rangerkms_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.8.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.8.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.8.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/HTTP_127.1.8.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 17:46:00 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271960, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 17:46:00.433524 1058 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 17:46:00.443816 1058 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 17:46:01.425391 1058 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 17:46:01.425478 1058 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms
I20260924 17:46:01.425498 1058 mini_ranger_kms.cc:227] Using postgres at 127.1.8.148:55045
2026-09-24 17:46:01,615 [I] DB FLAVOR :POSTGRES
2026-09-24 17:46:01,615 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 17:46:01,615 [I] Checking connection
2026-09-24 17:46:01,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 17:46:01,861 [I] connection success
2026-09-24 17:46:01,861 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 17:46:01,861 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 17:46:01,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 17:46:02,137 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 17:46:02,138 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 17:46:02,138 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 17:46:02,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.8.148:55045/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 17:46:02,419 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 17:46:02.935671 1058 mini_ranger_kms.cc:310] Created kms service
I20260924 17:46:03.166908 1058 mini_ranger_kms.cc:326] Created kudu user
I20260924 17:46:03.214924 1058 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 17:46:03 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271960, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:03.664773 1058 mini_ranger_kms.cc:384] Added ranger policy
I20260924 17:46:03.665140 1058 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms:/tmp/dist-test-taskkSMFuh/build/release/bin/postgresql.jar:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskkSMFuh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 17:46:03.665177 1058 mini_ranger_kms.cc:240] Using host: 127.1.8.148
I20260924 17:46:03.668808 1058 mini_ranger_kms.cc:276] Ranger KMS PID: 13240
I20260924 17:46:03.668906 1058 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.8.148:51095
Sep 24, 2026 5:46:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:46:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:46:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:46:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:46:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:46:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 17:46:06.021283 1058 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.588s user 0.007s sys 0.167s
I20260924 17:46:06.021399 1058 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 17:46:06.021447 1058 mini_ranger_kms.cc:401] 127.1.8.148:51095/kms/v1/keys
Sep 24 17:46:06 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271966, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.8.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 17:46:06 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271966, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.8.148@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
Sep 24 17:46:07 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271966, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.8.148@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
Sep 24 17:46:09 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:46:09.410053 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41955
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:41955
--ranger_config_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:09.493412 13290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:09.493637 13290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:09.493661 13290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:09.494975 13290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:09.495021 13290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:09.495036 13290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:09.495049 13290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:09.495062 13290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:09.496737 13290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.8.148:51095/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:41955
--ranger_config_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:09.497004 13290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:09.497273 13290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:09.502880 13295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:09.502871 13298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:09.502875 13290 server_base.cc:1034] running on GCE node
W20260924 17:46:09.503283 13296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:09.503583 13290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:09.503918 13290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:09.505111 13290 hybrid_clock.cc:648] HybridClock initialized: now 1790271969505089 us; error 64 us; skew 500 ppm
Sep 24 17:46:09 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:09.507754 13290 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:09.508232 13290 webserver.cc:460] Webserver started at http://127.1.8.190:43445/ using document root <none> and password file <none>
I20260924 17:46:09.508476 13290 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:09.508535 13290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:09.508698 13290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:46:09 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271969, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:09.555270 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "79f593a53b65498cb31ce1209a8bd83e"
format_stamp: "Formatted at 2026-09-24 17:46:09 on dist-test-slave-r7hf"
server_key: "d66a1e5bbbd10f6455e6c0ab50a74630"
server_key_iv: "679621b6de53964a4380a375b0a298d9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:09.555750 13290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "79f593a53b65498cb31ce1209a8bd83e"
format_stamp: "Formatted at 2026-09-24 17:46:09 on dist-test-slave-r7hf"
server_key: "d66a1e5bbbd10f6455e6c0ab50a74630"
server_key_iv: "679621b6de53964a4380a375b0a298d9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:09.557286 13290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:09.573354 13305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:09.573869 13290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:09.573994 13290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "79f593a53b65498cb31ce1209a8bd83e"
format_stamp: "Formatted at 2026-09-24 17:46:09 on dist-test-slave-r7hf"
server_key: "d66a1e5bbbd10f6455e6c0ab50a74630"
server_key_iv: "679621b6de53964a4380a375b0a298d9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:09.574095 13290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:09.605557 13290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:09.607334 13290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:09.607455 13290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:09.611581 13290 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:41955
I20260924 17:46:09.611622 13357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:41955 every 8 connection(s)
I20260924 17:46:09.612004 13290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:09.612673 13358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:09.615128 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Bootstrap starting.
I20260924 17:46:09.615731 13358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:09.616027 13358 log.cc:824] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:09.616350 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 13290
I20260924 17:46:09.616429 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:46:09.616626 1058 external_mini_cluster.cc:1428] Setting key fc40347191fb254e7fccea817a8d6c1a
I20260924 17:46:09.616840 13358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: No bootstrap required, opened a new log
I20260924 17:46:09.619730 13358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:09.619905 13358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:09.619939 13358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f593a53b65498cb31ce1209a8bd83e, State: Initialized, Role: FOLLOWER
I20260924 17:46:09.620062 13358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:09.620131 13358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:09.620159 13358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:09.620203 13358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:09.620891 13358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:09.621021 13358 leader_election.cc:302] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79f593a53b65498cb31ce1209a8bd83e; no voters:
I20260924 17:46:09.621225 13358 leader_election.cc:288] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:46:09.621253 13363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:09.621448 13363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 LEADER]: Becoming Leader. State: Replica: 79f593a53b65498cb31ce1209a8bd83e, State: Running, Role: LEADER
I20260924 17:46:09.621580 13363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:09.621608 13358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:09.621996 13365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79f593a53b65498cb31ce1209a8bd83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } } }
I20260924 17:46:09.622259 13365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:09.622187 13364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79f593a53b65498cb31ce1209a8bd83e. Latest consensus state: current_term: 1 leader_uuid: "79f593a53b65498cb31ce1209a8bd83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } } }
I20260924 17:46:09.622737 13364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:09.623577 13358 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 17:46:09 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:09.626129 13369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:09.627246 13369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:09.629690 13369 catalog_manager.cc:1347] Generated new cluster ID: 092886931c614e95949554dff1639e6c
I20260924 17:46:09.629735 13369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:46:09.631209 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:09.617196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.1.8.190:41955)
0924 17:46:09.617384 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:09.617395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:09.617431 (+ 36us) server_negotiation.cc:407] Connection header received
0924 17:46:09.618230 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:09.618240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:09.618309 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:09.618493 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:09.619070 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:09.620419 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:09.621292 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:09.621949 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:09.622438 (+ 489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:09.625602 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:09.625675 (+ 73us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:09.625675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:09.625781 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:09.628842 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:09.629333 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:09.629337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:09.629337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:09.629437 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:09.629649 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:09.629650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:09.629651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:09.629978 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:09.630138 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:09.630665 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 17:46:09.630979 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260924 17:46:09.637900 13369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:46:09.638538 13369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:09.642513 13369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Generated new TSK 0
I20260924 17:46:09.642728 13369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:46:10.333757 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41955
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:10.422873 13403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:10.423161 13403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:10.423198 13403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:10.425015 13403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:10.425083 13403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:10.425125 13403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:10.427420 13403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.8.148:51095/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:10.427753 13403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:10.428108 13403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:10.433450 13411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:10.433564 13409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:10.433744 13403 server_base.cc:1034] running on GCE node
W20260924 17:46:10.433447 13408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:10.434074 13403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:10.434350 13403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:10.435537 13403 hybrid_clock.cc:648] HybridClock initialized: now 1790271970435515 us; error 43 us; skew 500 ppm
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:10.438241 13403 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:10.438807 13403 webserver.cc:460] Webserver started at http://127.1.8.129:40535/ using document root <none> and password file <none>
I20260924 17:46:10.439059 13403 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:10.439127 13403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:10.439278 13403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:10.452565 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "0cf8ddd7f8c8439a851e979c4602b3ff"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "c18733020fec7fb5660eadcd262d00de"
server_key_iv: "aa7500a8034e719a3af144d5296d20d1"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.453045 13403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "0cf8ddd7f8c8439a851e979c4602b3ff"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "c18733020fec7fb5660eadcd262d00de"
server_key_iv: "aa7500a8034e719a3af144d5296d20d1"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.455853 13403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 17:46:10.469573 13418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:10.469893 13403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:10.470013 13403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "0cf8ddd7f8c8439a851e979c4602b3ff"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "c18733020fec7fb5660eadcd262d00de"
server_key_iv: "aa7500a8034e719a3af144d5296d20d1"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.470117 13403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:10.503515 13403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:10.506117 13403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:10.506364 13403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:10.506724 13403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:10.507170 13403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:10.507256 13403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:10.507311 13403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:10.507359 13403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:10.514034 13403 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:45233
I20260924 17:46:10.514564 13403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:10.520424 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 13403
I20260924 17:46:10.520534 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:46:10.520741 1058 external_mini_cluster.cc:1428] Setting key ebad192825c6559f4c2487e70c072af4
I20260924 17:46:10.522689 13533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:45233 every 8 connection(s)
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:46:10.536165 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:10.522271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:36347 (local address 127.1.8.190:41955)
0924 17:46:10.523380 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:10.523382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:10.523391 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:46:10.524165 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:10.524166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:10.524182 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:10.524228 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:10.524995 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.525528 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.528819 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.529149 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.529221 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:10.532160 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:10.532162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:10.532162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:10.532171 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:10.534158 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:10.534713 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:10.534714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:10.534714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:10.534782 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:10.535104 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:10.535105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:10.535105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:10.535264 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:10.535332 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:10.535903 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 17:46:10.536004 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":84,"threads_started":1}
I20260924 17:46:10.536854 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:10.522266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41955 (local address 127.1.8.129:36347)
0924 17:46:10.523253 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:10.523308 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:10.524035 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:10.524436 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:10.524441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:10.524690 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:10.524890 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:10.524895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.525642 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.525649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:10.526567 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:10.526570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.529307 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.529309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:10.529487 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:10.530126 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:10.530137 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:10.532026 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:10.534259 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:10.534268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:10.534271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:10.534626 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:10.534854 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:10.534854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:10.534855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:10.535019 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:10.535361 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:10.535367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:10.535566 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 17:46:10.535666 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 17:46:10.536599 (+ 933us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":140,"threads_started":1}
I20260924 17:46:10.537717 13534 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41955
I20260924 17:46:10.537812 13534 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:10.538051 13534 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:10.538687 13322 ts_manager.cc:194] Registered new tserver with Master: 0cf8ddd7f8c8439a851e979c4602b3ff (127.1.8.129:45233)
I20260924 17:46:10.539597 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:36347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:46:10.567378 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41955
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 17:46:10.688690 13540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:10.689076 13540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:10.689170 13540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:10.691280 13540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:10.691426 13540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:10.691516 13540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:10.694231 13540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.8.148:51095/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:10.694839 13540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:10.695209 13540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:10.701803 13550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:10.702188 13547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:10.702524 13548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:10.703382 13540 server_base.cc:1034] running on GCE node
I20260924 17:46:10.703719 13540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:10.704011 13540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:10.707301 13540 hybrid_clock.cc:648] HybridClock initialized: now 1790271970707264 us; error 53 us; skew 500 ppm
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:10.710813 13540 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:10.711357 13540 webserver.cc:460] Webserver started at http://127.1.8.130:34617/ using document root <none> and password file <none>
I20260924 17:46:10.711638 13540 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:10.711692 13540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:10.711824 13540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:10.726598 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "869d6052f08741de943d88ca54aa47eb"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "ef97fd67c07165e2a5904171c41fc208"
server_key_iv: "dafd28462342fcaaae52e2eec9b696f9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.727136 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "869d6052f08741de943d88ca54aa47eb"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "ef97fd67c07165e2a5904171c41fc208"
server_key_iv: "dafd28462342fcaaae52e2eec9b696f9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.730842 13540 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 17:46:10.754158 13557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:10.754480 13540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:10.754592 13540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "869d6052f08741de943d88ca54aa47eb"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "ef97fd67c07165e2a5904171c41fc208"
server_key_iv: "dafd28462342fcaaae52e2eec9b696f9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.754714 13540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:10.781919 13540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:10.784433 13540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:10.784600 13540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:10.784878 13540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:10.785257 13540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:10.785312 13540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:10.785398 13540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:10.785425 13540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:10.792826 13540 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:34045
I20260924 17:46:10.793078 13671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:34045 every 8 connection(s)
I20260924 17:46:10.793360 13540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:46:10.794031 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 13540
I20260924 17:46:10.794134 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:46:10.794341 1058 external_mini_cluster.cc:1428] Setting key c5bdd74dea5b4fc88fba6b5bee35e822
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:46:10.806316 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:10.794342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:44925 (local address 127.1.8.190:41955)
0924 17:46:10.794424 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:10.794427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:10.795611 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 17:46:10.796392 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:10.796393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:10.796405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:10.796440 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:10.797315 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.798132 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.799487 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.799815 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.799893 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:10.802794 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:10.802795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:10.802796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:10.802805 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:10.804575 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:10.805117 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:10.805118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:10.805119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:10.805181 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:10.805495 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:10.805496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:10.805497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:10.805642 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:10.805700 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:10.806101 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 17:46:10.806209 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:46:10.806952 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:10.794206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41955 (local address 127.1.8.130:44925)
0924 17:46:10.795477 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:10.795525 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:10.796260 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:10.796648 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:10.796652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:10.796920 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:10.797209 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:10.797214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.798251 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.798252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:10.799387 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:10.799389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:10.799993 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:10.799994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:10.800143 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:10.800768 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:10.800779 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:10.802664 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:10.804678 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:10.804686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:10.804689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:10.805018 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:10.805255 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:10.805256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:10.805256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:10.805403 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:10.805733 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:10.805739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:10.805933 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 17:46:10.806599 (+ 666us) client_negotiation.cc:240] Negotiation successful
0924 17:46:10.806770 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"thread_start_us":652,"threads_started":1}
I20260924 17:46:10.807535 13672 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41955
I20260924 17:46:10.807634 13672 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:10.807816 13672 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:10.808215 13322 ts_manager.cc:194] Registered new tserver with Master: 869d6052f08741de943d88ca54aa47eb (127.1.8.130:34045)
I20260924 17:46:10.808696 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:44925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:46:10.825831 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41955
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:10.956331 13677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:10.956701 13677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:10.956791 13677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:10.958853 13677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:10.958992 13677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:10.959077 13677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:10.961459 13677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.8.148:51095/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:10.962007 13677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:10.962349 13677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:10.969378 13685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:10.969422 13683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:10.969782 13682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:10.971010 13677 server_base.cc:1034] running on GCE node
I20260924 17:46:10.971298 13677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:10.971611 13677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:10.976257 13677 hybrid_clock.cc:648] HybridClock initialized: now 1790271970976216 us; error 38 us; skew 500 ppm
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:10.979898 13677 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:10.980528 13677 webserver.cc:460] Webserver started at http://127.1.8.131:41919/ using document root <none> and password file <none>
I20260924 17:46:10.980836 13677 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:10.980918 13677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:10.981137 13677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 17:46:10 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:10.998735 13677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "59036d6a8c98462483ad85415c4f08e3"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "38ee51994278cc5eae7dd26937e4b338"
server_key_iv: "1d6490883e649b3973715620e4334cac"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:10.999338 13677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "59036d6a8c98462483ad85415c4f08e3"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "38ee51994278cc5eae7dd26937e4b338"
server_key_iv: "1d6490883e649b3973715620e4334cac"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:11.002310 13677 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:11.017401 13692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.017674 13677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:11.017776 13677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "59036d6a8c98462483ad85415c4f08e3"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "38ee51994278cc5eae7dd26937e4b338"
server_key_iv: "1d6490883e649b3973715620e4334cac"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:11.017874 13677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:11.028971 13677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:11.031229 13677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:11.031402 13677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:11.031683 13677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:11.032084 13677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:11.032140 13677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.032207 13677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:11.032236 13677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.050292 13677 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:46583
I20260924 17:46:11.050907 13677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:46:11.051877 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:46583 every 8 connection(s)
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271970, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:11.059747 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 13677
I20260924 17:46:11.059839 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:46:11.060041 1058 external_mini_cluster.cc:1428] Setting key 12c47bb36852e6748457f8431dce9912
I20260924 17:46:11.064903 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.052522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:56621 (local address 127.1.8.190:41955)
0924 17:46:11.052571 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:11.052573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:11.053134 (+ 561us) server_negotiation.cc:407] Connection header received
0924 17:46:11.053818 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:11.053819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:11.053833 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:11.053867 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:11.054731 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.055516 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.056951 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.057262 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.057346 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.061388 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:11.061389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:11.061390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:11.061398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:11.063198 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.063766 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.063767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.063768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.063835 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.064125 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.064126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.064126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.064263 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:11.064318 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:11.064734 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 17:46:11.064832 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:46:11.065389 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.052516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41955 (local address 127.1.8.131:56621)
0924 17:46:11.052961 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:11.053013 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:11.053695 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:11.054057 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:11.054062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:11.054308 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:11.054625 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.054632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.055626 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.055627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.056841 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.056843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.057386 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.057387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.057522 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.058187 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:11.058198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:11.061259 (+ 3061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:11.063297 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.063306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.063309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.063670 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.063907 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.063908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.063908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.064051 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.064366 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:11.064383 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:11.064576 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 17:46:11.064695 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 17:46:11.064865 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":85,"threads_started":1}
I20260924 17:46:11.065973 13806 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41955
I20260924 17:46:11.066092 13806 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.066306 13806 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:11.066828 13322 ts_manager.cc:194] Registered new tserver with Master: 59036d6a8c98462483ad85415c4f08e3 (127.1.8.131:46583)
I20260924 17:46:11.067323 13322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:56621
I20260924 17:46:11.075075 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:46:11.075825 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 13403
I20260924 17:46:11.085132 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 13540
I20260924 17:46:11.093048 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 13677
I20260924 17:46:11.098448 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 13290
I20260924 17:46:11.103132 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41955
--webserver_interface=127.1.8.190
--webserver_port=43445
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:41955
--ranger_config_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:11.188072 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:11.188305 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:11.188364 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:11.189585 13809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:11.189635 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:11.189651 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:11.189662 13809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:11.189674 13809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:11.191443 13809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:41955
--ranger_config_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=43445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:11.191712 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:11.191967 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:11.197758 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.197758 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:11.198045 13809 server_base.cc:1034] running on GCE node
W20260924 17:46:11.197763 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:11.198450 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:11.198794 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:11.199986 13809 hybrid_clock.cc:648] HybridClock initialized: now 1790271971199966 us; error 36 us; skew 500 ppm
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:11.202710 13809 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:11.203253 13809 webserver.cc:460] Webserver started at http://127.1.8.190:43445/ using document root <none> and password file <none>
I20260924 17:46:11.203513 13809 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:11.203594 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:11.254806 13809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:11.255623 13825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.255782 13809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:11.255882 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "79f593a53b65498cb31ce1209a8bd83e"
format_stamp: "Formatted at 2026-09-24 17:46:09 on dist-test-slave-r7hf"
server_key: "d66a1e5bbbd10f6455e6c0ab50a74630"
server_key_iv: "679621b6de53964a4380a375b0a298d9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:11.256269 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:11.277971 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:11.280433 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:11.280627 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:11.285849 13809 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:41955
I20260924 17:46:11.285887 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:41955 every 8 connection(s)
I20260924 17:46:11.286362 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:11.287768 13878 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:46:11.288491 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Bootstrap starting.
I20260924 17:46:11.290129 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 13809
I20260924 17:46:11.290506 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:45233
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:41955
--webserver_port=40535
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:11.298759 13878 log.cc:824] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:11.301474 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:11.301852 13878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Bootstrap complete.
I20260924 17:46:11.305387 13878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:11.305737 13878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79f593a53b65498cb31ce1209a8bd83e, State: Initialized, Role: FOLLOWER
I20260924 17:46:11.305904 13878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:11.306010 13878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:11.306097 13878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:11.306172 13878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:46:11.307680 13878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:11.307848 13878 leader_election.cc:302] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79f593a53b65498cb31ce1209a8bd83e; no voters:
I20260924 17:46:11.308073 13878 leader_election.cc:288] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:46:11.308154 13882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:46:11.308395 13882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [term 2 LEADER]: Becoming Leader. State: Replica: 79f593a53b65498cb31ce1209a8bd83e, State: Running, Role: LEADER
I20260924 17:46:11.308406 13878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:11.308599 13882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } }
I20260924 17:46:11.308869 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "79f593a53b65498cb31ce1209a8bd83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } } }
I20260924 17:46:11.308936 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:11.309037 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79f593a53b65498cb31ce1209a8bd83e. Latest consensus state: current_term: 2 leader_uuid: "79f593a53b65498cb31ce1209a8bd83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79f593a53b65498cb31ce1209a8bd83e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41955 } } }
I20260924 17:46:11.309082 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:11.309626 13886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:11.310195 13886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:11.310374 13886 catalog_manager.cc:1259] Loaded cluster ID: 092886931c614e95949554dff1639e6c
I20260924 17:46:11.310488 13886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:46:11.311733 13886 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:11.311976 13886 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Loaded TSK: 0
I20260924 17:46:11.312260 13886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:46:11.313246 13878 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 17:46:11.410133 13880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:11.410460 13880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:11.410508 13880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:11.412451 13880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:11.412526 13880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:11.412580 13880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:11.415002 13880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:45233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=40535
--tserver_master_addrs=127.1.8.190:41955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:11.415277 13880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:11.415607 13880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:11.422528 13916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.422655 13915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.422555 13918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:11.423877 13880 server_base.cc:1034] running on GCE node
I20260924 17:46:11.424239 13880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:11.424561 13880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:11.425734 13880 hybrid_clock.cc:648] HybridClock initialized: now 1790271971425703 us; error 52 us; skew 500 ppm
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:11.429414 13880 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:11.430035 13880 webserver.cc:460] Webserver started at http://127.1.8.129:40535/ using document root <none> and password file <none>
I20260924 17:46:11.430330 13880 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:11.430444 13880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:11.456214 13880 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:11.457300 13926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.457523 13880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:11.457623 13880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "0cf8ddd7f8c8439a851e979c4602b3ff"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "c18733020fec7fb5660eadcd262d00de"
server_key_iv: "aa7500a8034e719a3af144d5296d20d1"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:11.458079 13880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:11.480268 13880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:11.483464 13880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:11.483726 13880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:11.484054 13880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:11.484532 13880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:11.484622 13880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.484726 13880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:11.484804 13880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.492496 13880 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:45233
I20260924 17:46:11.493134 13880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:11.493497 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 13880
I20260924 17:46:11.493785 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:34045
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:41955
--webserver_port=34617
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:11.494953 14039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:45233 every 8 connection(s)
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:11.511304 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.496857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:34025 (local address 127.1.8.190:41955)
0924 17:46:11.497212 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:11.497223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:11.497849 (+ 626us) server_negotiation.cc:407] Connection header received
0924 17:46:11.498793 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:11.498802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:11.498869 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:11.498990 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:11.499943 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.501398 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.503006 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.503653 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.503753 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.506716 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:11.506722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:11.506722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:11.506736 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:11.508893 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.509543 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.509548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.509548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.509672 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.509977 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.509978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.509978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.510420 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:11.510498 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:11.510909 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 17:46:11.511060 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":86,"threads_started":1}
I20260924 17:46:11.512207 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.496751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41955 (local address 127.1.8.129:34025)
0924 17:46:11.497621 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:11.497675 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:11.498511 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:11.499256 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:11.499260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:11.499505 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:11.499793 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.499800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.501540 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.501540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.502892 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.502895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.503911 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.503912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.504055 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.504744 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:11.504755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:11.506559 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:11.509081 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.509100 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.509103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.509430 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.509753 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.509754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.509755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.509891 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.510536 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:11.510542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:11.510706 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 17:46:11.511775 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0924 17:46:11.511960 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":90,"threads_started":1}
I20260924 17:46:11.513288 14041 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41955
I20260924 17:46:11.513386 14041 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.513584 14041 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
W20260924 17:46:11.514577 14041 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:41955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 17:46:11.514665 14041 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.514731 14041 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:11.515079 14041 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.515165 14041 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
W20260924 17:46:11.515548 14041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:46:11.611227 14040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:11.611594 14040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:11.611688 14040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:11.613583 14040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:11.613668 14040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:11.613715 14040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:11.616000 14040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:34045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=34617
--tserver_master_addrs=127.1.8.190:41955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:11.616506 14040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:11.616827 14040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:11.622675 14050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.622690 14049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.622704 14052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:11.623987 14040 server_base.cc:1034] running on GCE node
I20260924 17:46:11.624342 14040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:11.624642 14040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:11.625816 14040 hybrid_clock.cc:648] HybridClock initialized: now 1790271971625791 us; error 37 us; skew 500 ppm
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:11.629381 14040 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:11.630087 14040 webserver.cc:460] Webserver started at http://127.1.8.130:34617/ using document root <none> and password file <none>
I20260924 17:46:11.630379 14040 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:11.630496 14040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:11.648316 14040 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:11.649667 14060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.649902 14040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:11.650010 14040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "869d6052f08741de943d88ca54aa47eb"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "ef97fd67c07165e2a5904171c41fc208"
server_key_iv: "dafd28462342fcaaae52e2eec9b696f9"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:11.650699 14040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:11.668102 14040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:11.670336 14040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:11.670576 14040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:11.670869 14040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:11.671274 14040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:11.671330 14040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.671398 14040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:11.671428 14040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.677985 14040 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:34045
I20260924 17:46:11.678169 14173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:34045 every 8 connection(s)
I20260924 17:46:11.678666 14040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:46:11.680851 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14040
I20260924 17:46:11.681217 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.8.148:51095/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:46583
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:41955
--webserver_port=41919
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:11.693228 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.680283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:40947 (local address 127.1.8.190:41955)
0924 17:46:11.680338 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:11.680341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:11.680621 (+ 280us) server_negotiation.cc:407] Connection header received
0924 17:46:11.681902 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:11.681903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:11.681923 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:11.681954 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:11.683069 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.683880 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.685502 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.685858 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.685928 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.689135 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:11.689137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:11.689138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:11.689148 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:11.691198 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.691830 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.691832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.691833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.691905 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.692255 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.692256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.692257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.692428 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:11.692504 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:11.692956 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 17:46:11.693084 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:46:11.694037 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.680314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41955 (local address 127.1.8.130:40947)
0924 17:46:11.680463 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:11.680515 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:11.681739 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:11.682153 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:11.682158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:11.682440 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:11.682961 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.682968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.684003 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.684005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.685389 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.685392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.686123 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.686125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.686289 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.686998 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:11.687022 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:11.688976 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:11.691323 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.691331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.691335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.691725 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.691999 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.692000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.692001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.692174 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.692538 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:11.692546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:11.692767 (+ 221us) client_negotiation.cc:769] Sending connection context
0924 17:46:11.693491 (+ 724us) client_negotiation.cc:240] Negotiation successful
0924 17:46:11.693706 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260924 17:46:11.694540 14174 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41955
I20260924 17:46:11.694676 14174 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.694869 14174 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
W20260924 17:46:11.695639 14174 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:41955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 17:46:11.695744 14174 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.695808 14174 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:11.696152 14174 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.696226 14174 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
W20260924 17:46:11.696535 14174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:46:11.793337 14177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:11.793723 14177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:11.793779 14177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:11.796708 14177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:11.796826 14177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:11.796883 14177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:11.801124 14177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.8.148:51095/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:46583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=41919
--tserver_master_addrs=127.1.8.190:41955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:11.801451 14177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:11.801734 14177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:46:11.809859 14177 server_base.cc:1034] running on GCE node
W20260924 17:46:11.809872 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.809883 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:11.816706 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:11.817104 14177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:11.817466 14177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:11.818666 14177 hybrid_clock.cc:648] HybridClock initialized: now 1790271971818651 us; error 33 us; skew 500 ppm
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:11.821615 14177 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:11.822180 14177 webserver.cc:460] Webserver started at http://127.1.8.131:41919/ using document root <none> and password file <none>
I20260924 17:46:11.822515 14177 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:11.822611 14177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:11.850442 14177 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 17:46:11.851482 14194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.851856 14177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:11.851959 14177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "59036d6a8c98462483ad85415c4f08e3"
format_stamp: "Formatted at 2026-09-24 17:46:10 on dist-test-slave-r7hf"
server_key: "38ee51994278cc5eae7dd26937e4b338"
server_key_iv: "1d6490883e649b3973715620e4334cac"
server_key_version: "kuduclusterkey@0"
I20260924 17:46:11.852391 14177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:11.866948 14177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:11.869431 14177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:11.869655 14177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:11.869979 14177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:11.870448 14177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:11.870532 14177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.870600 14177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:11.870647 14177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:11.877842 14177 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:46583
I20260924 17:46:11.877919 14307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:46583 every 8 connection(s)
I20260924 17:46:11.878262 14177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:46:11.883417 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14177
Sep 24 17:46:11 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271971, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:11.889029 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.879139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:50659 (local address 127.1.8.190:41955)
0924 17:46:11.879219 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:11.879221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:11.879474 (+ 253us) server_negotiation.cc:407] Connection header received
0924 17:46:11.880025 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:11.880026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:11.880039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:11.880102 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:11.880906 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.881399 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.882956 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.883149 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.883261 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.885775 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:11.885777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:11.885777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:11.885786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:11.887441 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.887868 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.887870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.887870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.887935 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.888267 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.888268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.888269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.888401 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:11.888468 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:11.888836 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 17:46:11.888925 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:46:11.889276 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.879088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41955 (local address 127.1.8.131:50659)
0924 17:46:11.879358 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:11.879390 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:11.879908 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:11.880232 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:11.880238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:11.880495 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:11.880791 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.880798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.881505 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.881507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.882865 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:11.882869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.883361 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.883363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:11.883544 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.884161 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:11.884175 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:11.885655 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:11.887525 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.887532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.887534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.887770 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.888008 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:11.888009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:11.888009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:11.888186 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:11.888462 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:11.888471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:11.888671 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 17:46:11.888805 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 17:46:11.888986 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260924 17:46:11.889921 14308 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41955
I20260924 17:46:11.890025 14308 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.890209 14308 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
W20260924 17:46:11.890859 14308 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:41955 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 17:46:11.890938 14308 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.891000 14308 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:11.891395 14308 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:11.891460 14308 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
W20260924 17:46:11.891808 14308 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:46:11.892271 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:11.883880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.1.8.190:41955)
0924 17:46:11.884069 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:11.884071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:11.884206 (+ 135us) server_negotiation.cc:407] Connection header received
0924 17:46:11.884318 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:11.884320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:11.884331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:11.884376 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:11.885804 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.886524 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.887540 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:11.887855 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:11.887923 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:11.889470 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:11.889471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:11.889472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:11.889478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:11.890953 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.891382 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.891384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.891384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.891445 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:11.891647 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:11.891648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:11.891648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:11.891834 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:11.892003 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:11.892083 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 17:46:11.892169 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
Sep 24 17:46:12 dist-test-slave-r7hf krb5kdc[12846](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790271972, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:12.515763 14041 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:12.515951 14041 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:12.516377 13842 ts_manager.cc:194] Registered new tserver with Master: 0cf8ddd7f8c8439a851e979c4602b3ff (127.1.8.129:45233)
I20260924 17:46:12.517508 13842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:34025
I20260924 17:46:12.696776 14174 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:12.696982 14174 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:12.697285 13841 ts_manager.cc:194] Registered new tserver with Master: 869d6052f08741de943d88ca54aa47eb (127.1.8.130:34045)
I20260924 17:46:12.697700 13841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:40947
Sep 24 17:46:12 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271972, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
I20260924 17:46:12.892041 14308 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:12.892220 14308 heartbeater.cc:511] Master 127.1.8.190:41955 requested a full tablet report, sending...
I20260924 17:46:12.892575 13841 ts_manager.cc:194] Registered new tserver with Master: 59036d6a8c98462483ad85415c4f08e3 (127.1.8.131:46583)
I20260924 17:46:12.893146 13841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:50659
I20260924 17:46:12.893184 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:46:12.911850 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.894108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.1.8.190:41955)
0924 17:46:12.894361 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:12.894364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:12.894551 (+ 187us) server_negotiation.cc:407] Connection header received
0924 17:46:12.894575 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:12.894575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:12.894586 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:12.894644 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:12.895362 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.896144 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.897197 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.897510 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.897573 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.907593 (+ 10020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:12.907595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:12.907595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:12.907606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:12.909922 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:12.910520 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:12.910521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:12.910522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:12.910593 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:12.910816 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:12.910817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:12.910817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:12.910972 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:12.911087 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:12.911625 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 17:46:12.911718 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":63,"threads_started":1}
I20260924 17:46:12.914373 13841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:46:12.916736 13841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:46:12.944322 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.926858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:46583 (local address 127.0.0.1:39596)
0924 17:46:12.934701 (+ 7843us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:12.934771 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:12.934879 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:12.935779 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:12.935782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:12.935784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:12.935889 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:12.935891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.937252 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.937253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:12.942260 (+ 5007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:12.942262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.943949 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.943950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:12.944055 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.944070 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:46:12.944150 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:46:12.944211 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7798,"thread_start_us":89,"threads_started":1}
I20260924 17:46:12.944681 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.927350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.1.8.131:46583)
0924 17:46:12.934586 (+ 7236us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:12.934593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:12.935277 (+ 684us) server_negotiation.cc:407] Connection header received
0924 17:46:12.935561 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:12.935564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:12.935583 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:12.935658 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:12.935985 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.937132 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.942360 (+ 5228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.943740 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.943828 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.943906 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:12.944544 (+ 638us) server_negotiation.cc:299] Negotiation successful
0924 17:46:12.944597 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7179,"thread_start_us":71,"threads_started":1}
I20260924 17:46:12.945657 14242 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7926a9ba6443ef93ef56923047d5b7 (DEFAULT_TABLE table=test-table [id=59f8a799ff484444ae5b81a3a079324c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:12.946049 14242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7926a9ba6443ef93ef56923047d5b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:12.946288 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.927134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:45233 (local address 127.0.0.1:50948)
0924 17:46:12.935362 (+ 8228us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:12.935373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:12.935422 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:12.939156 (+ 3734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:12.939158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:12.939161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:12.939209 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:12.939210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.940749 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.940750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:12.941965 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:12.941969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.946026 (+ 4057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.946028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:12.946100 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.946105 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:46:12.946161 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:46:12.946207 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8205,"thread_start_us":62,"threads_started":1}
I20260924 17:46:12.946642 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.927436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.1.8.129:45233)
0924 17:46:12.938888 (+ 11452us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:12.938894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:12.938907 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:46:12.938946 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:12.938948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:12.938967 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:12.939055 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:12.939294 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.940622 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.945040 (+ 4418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.945831 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.945926 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.945984 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:12.946505 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 17:46:12.946555 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11396,"thread_start_us":68,"threads_started":1}
I20260924 17:46:12.947331 13973 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7926a9ba6443ef93ef56923047d5b7 (DEFAULT_TABLE table=test-table [id=59f8a799ff484444ae5b81a3a079324c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:12.947710 13973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7926a9ba6443ef93ef56923047d5b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:12.953200 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.927619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:34045 (local address 127.0.0.1:41966)
0924 17:46:12.938481 (+ 10862us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:12.938494 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:12.941426 (+ 2932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:12.941821 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:12.941822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:12.941824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:12.941882 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:12.941883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.944819 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.944820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:12.951827 (+ 7007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:12.951832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.952939 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.952940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:12.953007 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.953012 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:46:12.953065 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:46:12.953114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10841,"thread_start_us":77,"threads_started":1}
I20260924 17:46:12.953846 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:12.927791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.1.8.130:34045)
0924 17:46:12.941593 (+ 13802us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:12.941599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:12.941609 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:46:12.941645 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:12.941647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:12.941663 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:12.941734 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:12.941958 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.943100 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.951967 (+ 8867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:12.952734 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:12.952817 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:12.952894 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:12.953623 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 17:46:12.953766 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13761,"thread_start_us":81,"threads_started":1}
I20260924 17:46:12.954779 14107 tablet_service.cc:1467] Processing CreateTablet for tablet 6d7926a9ba6443ef93ef56923047d5b7 (DEFAULT_TABLE table=test-table [id=59f8a799ff484444ae5b81a3a079324c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:12.954967 14341 tablet_bootstrap.cc:492] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Bootstrap starting.
I20260924 17:46:12.955193 14107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d7926a9ba6443ef93ef56923047d5b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:12.956557 14341 tablet_bootstrap.cc:654] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:12.959079 14341 log.cc:824] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:12.960098 14343 tablet_bootstrap.cc:492] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Bootstrap starting.
I20260924 17:46:12.960969 14343 tablet_bootstrap.cc:654] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:12.961324 14341 tablet_bootstrap.cc:492] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: No bootstrap required, opened a new log
I20260924 17:46:12.961371 14343 log.cc:824] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:12.961452 14341 ts_tablet_manager.cc:1397] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 17:46:12.962373 14343 tablet_bootstrap.cc:492] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: No bootstrap required, opened a new log
I20260924 17:46:12.962458 14343 ts_tablet_manager.cc:1397] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:12.964937 14341 raft_consensus.cc:348] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:12.965133 14341 raft_consensus.cc:374] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:12.965049 14343 raft_consensus.cc:348] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:12.965170 14343 raft_consensus.cc:374] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:12.965169 14341 raft_consensus.cc:729] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59036d6a8c98462483ad85415c4f08e3, State: Initialized, Role: FOLLOWER
I20260924 17:46:12.965199 14343 raft_consensus.cc:729] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869d6052f08741de943d88ca54aa47eb, State: Initialized, Role: FOLLOWER
I20260924 17:46:12.965528 14343 consensus_queue.cc:260] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:12.965931 14343 ts_tablet_manager.cc:1428] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:46:12.966095 14174 heartbeater.cc:503] Master 127.1.8.190:41955 was elected leader, sending a full tablet report...
I20260924 17:46:12.970571 14341 consensus_queue.cc:260] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:12.971099 14341 ts_tablet_manager.cc:1428] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20260924 17:46:12.971292 14308 heartbeater.cc:503] Master 127.1.8.190:41955 was elected leader, sending a full tablet report...
I20260924 17:46:12.972268 14342 tablet_bootstrap.cc:492] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: Bootstrap starting.
I20260924 17:46:12.973737 14342 tablet_bootstrap.cc:654] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:12.974732 14342 log.cc:824] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:12.976150 14342 tablet_bootstrap.cc:492] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: No bootstrap required, opened a new log
I20260924 17:46:12.976234 14342 ts_tablet_manager.cc:1397] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 17:46:12.980001 14342 raft_consensus.cc:348] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:12.980341 14342 raft_consensus.cc:374] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:12.980437 14342 raft_consensus.cc:729] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf8ddd7f8c8439a851e979c4602b3ff, State: Initialized, Role: FOLLOWER
I20260924 17:46:12.980644 14342 consensus_queue.cc:260] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:12.981106 14342 ts_tablet_manager.cc:1428] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260924 17:46:12.981294 14041 heartbeater.cc:503] Master 127.1.8.190:41955 was elected leader, sending a full tablet report...
W20260924 17:46:12.998119 14042 tablet.cc:2369] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 17:46:13 dist-test-slave-r7hf krb5kdc[12846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271972, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for HTTP/127.1.8.148@KRBTEST.COM
W20260924 17:46:13.131170 14309 tablet.cc:2369] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:46:13.181094 14175 tablet.cc:2369] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:46:13.327291 14346 raft_consensus.cc:482] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:13.327456 14346 raft_consensus.cc:504] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:13.327877 14346 leader_election.cc:288] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59036d6a8c98462483ad85415c4f08e3 (127.1.8.131:46583), 0cf8ddd7f8c8439a851e979c4602b3ff (127.1.8.129:45233)
I20260924 17:46:13.336701 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:13.328520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33033 (local address 127.1.8.131:46583)
0924 17:46:13.328585 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:13.328588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:13.330457 (+ 1869us) server_negotiation.cc:407] Connection header received
0924 17:46:13.330576 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:13.330577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:13.330591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:13.330621 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:13.330835 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.331592 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.333617 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.334209 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.334258 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:13.334270 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:13.336564 (+ 2294us) server_negotiation.cc:299] Negotiation successful
0924 17:46:13.336618 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:46:13.337204 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:13.328197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:46583 (local address 127.1.8.130:33033)
0924 17:46:13.330343 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:13.330357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:13.330494 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:13.330686 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:13.330687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:13.330688 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:13.330767 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:13.330769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.331699 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.331700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:13.333021 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:13.333025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.335173 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.335174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:13.335249 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:13.335253 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:13.337069 (+ 1816us) client_negotiation.cc:240] Negotiation successful
0924 17:46:13.337115 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"thread_start_us":92,"threads_started":1}
I20260924 17:46:13.337759 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7926a9ba6443ef93ef56923047d5b7" candidate_uuid: "869d6052f08741de943d88ca54aa47eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59036d6a8c98462483ad85415c4f08e3" is_pre_election: true
I20260924 17:46:13.337984 14259 raft_consensus.cc:2454] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869d6052f08741de943d88ca54aa47eb in term 0.
I20260924 17:46:13.338615 14062 leader_election.cc:302] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59036d6a8c98462483ad85415c4f08e3, 869d6052f08741de943d88ca54aa47eb; no voters:
I20260924 17:46:13.338763 14346 raft_consensus.cc:2781] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:46:13.338801 14346 raft_consensus.cc:482] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:46:13.338822 14346 raft_consensus.cc:3037] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:13.339720 14346 raft_consensus.cc:504] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:13.339866 14346 leader_election.cc:288] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [CANDIDATE]: Term 1 election: Requested vote from peers 59036d6a8c98462483ad85415c4f08e3 (127.1.8.131:46583), 0cf8ddd7f8c8439a851e979c4602b3ff (127.1.8.129:45233)
I20260924 17:46:13.340253 14259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7926a9ba6443ef93ef56923047d5b7" candidate_uuid: "869d6052f08741de943d88ca54aa47eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59036d6a8c98462483ad85415c4f08e3"
I20260924 17:46:13.340337 14259 raft_consensus.cc:3037] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:13.341212 14259 raft_consensus.cc:2454] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869d6052f08741de943d88ca54aa47eb in term 1.
I20260924 17:46:13.341454 14062 leader_election.cc:302] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59036d6a8c98462483ad85415c4f08e3, 869d6052f08741de943d88ca54aa47eb; no voters:
I20260924 17:46:13.341571 14346 raft_consensus.cc:2781] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:13.341720 14346 raft_consensus.cc:686] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 1 LEADER]: Becoming Leader. State: Replica: 869d6052f08741de943d88ca54aa47eb, State: Running, Role: LEADER
I20260924 17:46:13.341811 14346 consensus_queue.cc:237] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } }
I20260924 17:46:13.342702 13841 catalog_manager.cc:5495] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 869d6052f08741de943d88ca54aa47eb (127.1.8.130). New cstate: current_term: 1 leader_uuid: "869d6052f08741de943d88ca54aa47eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869d6052f08741de943d88ca54aa47eb" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 34045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 } health_report { overall_health: UNKNOWN } } }
I20260924 17:46:13.344029 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:13.328748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:57889 (local address 127.1.8.129:45233)
0924 17:46:13.328789 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:13.328791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:13.333220 (+ 4429us) server_negotiation.cc:407] Connection header received
0924 17:46:13.333322 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:13.333323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:13.333337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:13.333365 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:13.334335 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.335044 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.337635 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.338213 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.338259 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:13.338269 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:13.343880 (+ 5611us) server_negotiation.cc:299] Negotiation successful
0924 17:46:13.343948 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:46:13.344486 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:13.328681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:45233 (local address 127.1.8.130:57889)
0924 17:46:13.333160 (+ 4479us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:13.333172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:13.333250 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:13.333424 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:13.333424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:13.333426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:13.333529 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:13.333530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.335334 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.335334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:13.336458 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:13.336461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.338330 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.338331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:13.343780 (+ 5449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:13.343788 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 17:46:13.344352 (+ 564us) client_negotiation.cc:240] Negotiation successful
0924 17:46:13.344407 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4460,"thread_start_us":82,"threads_started":1}
I20260924 17:46:13.344992 13994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7926a9ba6443ef93ef56923047d5b7" candidate_uuid: "869d6052f08741de943d88ca54aa47eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" is_pre_election: true
I20260924 17:46:13.345157 13994 raft_consensus.cc:2454] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869d6052f08741de943d88ca54aa47eb in term 0.
I20260924 17:46:13.345284 13993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d7926a9ba6443ef93ef56923047d5b7" candidate_uuid: "869d6052f08741de943d88ca54aa47eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff"
I20260924 17:46:13.345346 13993 raft_consensus.cc:3037] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:13.346247 13993 raft_consensus.cc:2454] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869d6052f08741de943d88ca54aa47eb in term 1.
I20260924 17:46:13.374783 14340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:13.371178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.1.8.130:34045)
0924 17:46:13.371229 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:13.371232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:13.371245 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:46:13.371316 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:13.371317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:13.371342 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:13.371435 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:13.371612 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.372402 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.373582 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:13.373886 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:13.373947 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:13.374170 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:13.374314 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:13.374418 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:13.374674 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 17:46:13.374698 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:46:13.377070 13993 raft_consensus.cc:1261] T 6d7926a9ba6443ef93ef56923047d5b7 P 0cf8ddd7f8c8439a851e979c4602b3ff [term 1 FOLLOWER]: Refusing update from remote peer 869d6052f08741de943d88ca54aa47eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:13.377528 14354 consensus_queue.cc:1035] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf8ddd7f8c8439a851e979c4602b3ff" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:13.380059 14259 raft_consensus.cc:1261] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 1 FOLLOWER]: Refusing update from remote peer 869d6052f08741de943d88ca54aa47eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:13.380436 14354 consensus_queue.cc:1035] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "59036d6a8c98462483ad85415c4f08e3" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:13.385046 14357 mvcc.cc:205] Tried to move back new op lower bound from 7332954002949079040 to 7332954002808532992. Current Snapshot: MvccSnapshot[applied={T|T < 7332954002949079040}]
I20260924 17:46:13.385303 14361 mvcc.cc:205] Tried to move back new op lower bound from 7332954002949079040 to 7332954002808532992. Current Snapshot: MvccSnapshot[applied={T|T < 7332954002949079040}]
I20260924 17:46:13.387928 14358 mvcc.cc:205] Tried to move back new op lower bound from 7332954002949079040 to 7332954002808532992. Current Snapshot: MvccSnapshot[applied={T|T < 7332954002949079040}]
I20260924 17:46:13.391016 13841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:13.391124 13841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:13.391939 13841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79f593a53b65498cb31ce1209a8bd83e: Sending DeleteTablet for 3 replicas of tablet 6d7926a9ba6443ef93ef56923047d5b7
I20260924 17:46:13.392547 14107 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7926a9ba6443ef93ef56923047d5b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:13 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:41966
I20260924 17:46:13.392782 13973 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7926a9ba6443ef93ef56923047d5b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:13 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:50948
I20260924 17:46:13.392830 14242 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d7926a9ba6443ef93ef56923047d5b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:13 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:39596
I20260924 17:46:13.393390 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 13880
I20260924 17:46:13.406203 14368 tablet_replica.cc:330] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: stopping tablet replica
I20260924 17:46:13.406477 14368 raft_consensus.cc:2229] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:46:13.406754 14368 raft_consensus.cc:2258] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:13.407397 14368 ts_tablet_manager.cc:1905] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:46:13.409394 13829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:45233 (error 108)
I20260924 17:46:13.409592 14368 ts_tablet_manager.cc:1918] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:13.409664 14368 log.cc:1196] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/6d7926a9ba6443ef93ef56923047d5b7
I20260924 17:46:13.409934 14368 ts_tablet_manager.cc:1939] T 6d7926a9ba6443ef93ef56923047d5b7 P 869d6052f08741de943d88ca54aa47eb: Deleting consensus metadata
I20260924 17:46:13.410319 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:13.410192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:45233 (local address 127.0.0.1:50954)
0924 17:46:13.410237 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:13.410278 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:45233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 17:46:13.410535 13829 catalog_manager.cc:4584] TS 0cf8ddd7f8c8439a851e979c4602b3ff (127.1.8.129:45233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d7926a9ba6443ef93ef56923047d5b7: Network error: Client connection negotiation failed: client connection to 127.1.8.129:45233: connect: Connection refused (error 111)
I20260924 17:46:13.410974 13828 catalog_manager.cc:4841] TS 869d6052f08741de943d88ca54aa47eb (127.1.8.130:34045): tablet 6d7926a9ba6443ef93ef56923047d5b7 (table test-table [id=59f8a799ff484444ae5b81a3a079324c]) successfully deleted
I20260924 17:46:13.411082 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14040
I20260924 17:46:13.412333 14370 tablet_replica.cc:330] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: stopping tablet replica
I20260924 17:46:13.412478 14370 raft_consensus.cc:2229] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:13.412614 14370 raft_consensus.cc:2258] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:13.413175 14370 ts_tablet_manager.cc:1905] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:13.415284 14370 ts_tablet_manager.cc:1918] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:13.415421 14370 log.cc:1196] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/6d7926a9ba6443ef93ef56923047d5b7
I20260924 17:46:13.415759 14370 ts_tablet_manager.cc:1939] T 6d7926a9ba6443ef93ef56923047d5b7 P 59036d6a8c98462483ad85415c4f08e3: Deleting consensus metadata
I20260924 17:46:13.416546 13827 catalog_manager.cc:4841] TS 59036d6a8c98462483ad85415c4f08e3 (127.1.8.131:46583): tablet 6d7926a9ba6443ef93ef56923047d5b7 (table test-table [id=59f8a799ff484444ae5b81a3a079324c]) successfully deleted
I20260924 17:46:13.417615 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14177
I20260924 17:46:13.423925 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 13809
I20260924 17:46:13.430908 1058 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 17:46:13.775842 1058 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 17:46:13.775987 1058 mini_ranger.cc:66] Stopping Ranger...
I20260924 17:46:14.121155 1058 mini_ranger.cc:68] Stopped Ranger
2026-09-24 17:46:14.121 UTC [12876] LOG: received smart shutdown request
2026-09-24 17:46:14.125 UTC [12876] LOG: background worker "logical replication launcher" (PID 12885) exited with exit code 1
2026-09-24 17:46:14.126 UTC [12880] LOG: shutting down
2026-09-24 17:46:14.313 UTC [12876] LOG: database system is shut down
2026-09-24T17:46:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50961 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:46:14 dist-test-slave-r7hf krb5kdc[14385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:46:14 dist-test-slave-r7hf krb5kdc[14385](info): set up 2 sockets
Sep 24 17:46:14 dist-test-slave-r7hf krb5kdc[14385](info): commencing operation
krb5kdc: starting...
W20260924 17:46:15.506352 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:46:15 dist-test-slave-r7hf krb5kdc[14385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:46:15.633934 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39033
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39033,127.1.8.189:45133,127.1.8.188:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:15.717087 14398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:15.717386 14398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:15.717428 14398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:15.718711 14398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:15.718793 14398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:15.718818 14398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:15.718839 14398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:15.718870 14398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:15.720503 14398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39033,127.1.8.189:45133,127.1.8.188:43091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:15.720810 14398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:15.721063 14398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:15.726289 14406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:15.726260 14403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:15.726287 14404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:15.726512 14398 server_base.cc:1034] running on GCE node
I20260924 17:46:15.726871 14398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:15.727145 14398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:15.728291 14398 hybrid_clock.cc:648] HybridClock initialized: now 1790271975728265 us; error 38 us; skew 500 ppm
Sep 24 17:46:15 dist-test-slave-r7hf krb5kdc[14385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:15.730906 14398 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:15.731369 14398 webserver.cc:460] Webserver started at http://127.1.8.190:33891/ using document root <none> and password file <none>
I20260924 17:46:15.731606 14398 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:15.731652 14398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:15.731808 14398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:15.732946 14398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "361b0d0419ce47d4a094d68e37af9b05"
format_stamp: "Formatted at 2026-09-24 17:46:15 on dist-test-slave-r7hf"
server_key: "343ee75cf60b32d03529224ddaeada4b"
server_key_iv: "b6df3ba3e5575396a86b95165d61ba41"
server_key_version: "encryptionkey@0"
I20260924 17:46:15.733328 14398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "361b0d0419ce47d4a094d68e37af9b05"
format_stamp: "Formatted at 2026-09-24 17:46:15 on dist-test-slave-r7hf"
server_key: "343ee75cf60b32d03529224ddaeada4b"
server_key_iv: "b6df3ba3e5575396a86b95165d61ba41"
server_key_version: "encryptionkey@0"
I20260924 17:46:15.734862 14398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:46:15.735620 14413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:15.735809 14398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:46:15.735908 14398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "361b0d0419ce47d4a094d68e37af9b05"
format_stamp: "Formatted at 2026-09-24 17:46:15 on dist-test-slave-r7hf"
server_key: "343ee75cf60b32d03529224ddaeada4b"
server_key_iv: "b6df3ba3e5575396a86b95165d61ba41"
server_key_version: "encryptionkey@0"
I20260924 17:46:15.735986 14398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:15.786614 14398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:15.788821 14398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:15.788981 14398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:15.793141 14398 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39033
I20260924 17:46:15.793236 14465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39033 every 8 connection(s)
I20260924 17:46:15.793593 14398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:15.794602 14466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:15.796195 14466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:15.799774 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14398
I20260924 17:46:15.799906 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:46:15.800105 1058 external_mini_cluster.cc:1428] Setting key 1e14cd76dc2118fa1f030867f0c0f061
Sep 24 17:46:15 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:15.806747 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.796910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39033 (local address 127.0.0.1:38518)
0924 17:46:15.797160 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.797207 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:15.797738 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:15.798090 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:15.798092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:15.798452 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:15.798632 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:15.798637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.799698 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.799700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:15.800547 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:15.800550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.801076 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.801077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:15.801141 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:15.801745 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:15.801754 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:15.803229 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:15.805147 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:15.805154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:15.805157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:15.805368 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:15.805634 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:15.805635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:15.805636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:15.805730 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:15.806101 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:15.806105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:15.806169 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 17:46:15.806284 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 17:46:15.806558 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":120,"thread_start_us":115,"threads_started":1}
I20260924 17:46:15.806747 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.796974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.1.8.190:39033)
0924 17:46:15.797170 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:15.797181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:15.797317 (+ 136us) server_negotiation.cc:407] Connection header received
0924 17:46:15.797962 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:15.797963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:15.797981 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:15.798083 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:15.798738 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.799595 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.800640 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.800966 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.801101 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:15.803346 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:15.803354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:15.803354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:15.803361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:15.805026 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:15.805469 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:15.805472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:15.805473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:15.805564 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:15.805798 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:15.805798 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:15.805799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:15.806023 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:15.806125 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:15.806310 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 17:46:15.806483 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.1.8.189@KRBTEST.COM; defaulting to no policy
I20260924 17:46:15.807593 14466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:15.807976 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.807860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54802)
0924 17:46:15.807910 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.807954 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:46:15.808169 14414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
I20260924 17:46:15.808627 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.808523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54818)
0924 17:46:15.808584 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.808609 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 17:46:15.808753 14466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:45133: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab.
Entry for principal kudu/127.1.8.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab.
I20260924 17:46:15.825820 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:45133
--webserver_interface=127.1.8.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab
--principal=kudu/127.1.8.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39033,127.1.8.189:45133,127.1.8.188:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:15.858004 14466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } attempt: 1
I20260924 17:46:15.858646 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.858472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54828)
0924 17:46:15.858537 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.858610 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 17:46:15.859026 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.858929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54832)
0924 17:46:15.858979 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.859007 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:46:15.859130 14466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:45133: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
W20260924 17:46:15.908404 14473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:15.908622 14473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:15.908645 14473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:15.909739 14473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:15.909771 14473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:15.909785 14473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:15.909798 14473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:15.909811 14473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:15.911331 14473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39033,127.1.8.189:45133,127.1.8.188:43091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab
--principal=kudu/127.1.8.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:45133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.8.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:15.911540 14473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:15.911715 14473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:15.916738 14481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:15.916793 14479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:15.916829 14473 server_base.cc:1034] running on GCE node
W20260924 17:46:15.916906 14478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:15.917296 14473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:15.917579 14473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:15.918779 14473 hybrid_clock.cc:648] HybridClock initialized: now 1790271975918763 us; error 26 us; skew 500 ppm
Sep 24 17:46:15 dist-test-slave-r7hf krb5kdc[14385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:15.921722 14473 init.cc:377] Logged in from keytab as kudu/127.1.8.189@KRBTEST.COM (short username kudu)
I20260924 17:46:15.922161 14473 webserver.cc:460] Webserver started at http://127.1.8.189:36383/ using document root <none> and password file <none>
I20260924 17:46:15.922508 14473 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:15.922587 14473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:15.922690 14473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:15.923774 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data/instance:
uuid: "ea57cf7b45c042ed8fb6dad33097d333"
format_stamp: "Formatted at 2026-09-24 17:46:15 on dist-test-slave-r7hf"
server_key: "8020b171167e94dc6aeb2b9513023f41"
server_key_iv: "e116ae3cdd2742081eb2052f74b12433"
server_key_version: "encryptionkey@0"
I20260924 17:46:15.924096 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal/instance:
uuid: "ea57cf7b45c042ed8fb6dad33097d333"
format_stamp: "Formatted at 2026-09-24 17:46:15 on dist-test-slave-r7hf"
server_key: "8020b171167e94dc6aeb2b9513023f41"
server_key_iv: "e116ae3cdd2742081eb2052f74b12433"
server_key_version: "encryptionkey@0"
I20260924 17:46:15.925473 14473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:46:15.926179 14488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:15.926353 14473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:15.926458 14473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal
uuid: "ea57cf7b45c042ed8fb6dad33097d333"
format_stamp: "Formatted at 2026-09-24 17:46:15 on dist-test-slave-r7hf"
server_key: "8020b171167e94dc6aeb2b9513023f41"
server_key_iv: "e116ae3cdd2742081eb2052f74b12433"
server_key_version: "encryptionkey@0"
I20260924 17:46:15.926527 14473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:15.927354 14466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } attempt: 2
I20260924 17:46:15.927806 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.927673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54844)
0924 17:46:15.927736 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.927768 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 17:46:15.928206 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.928120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54858)
0924 17:46:15.928166 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.928191 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:46:15.928318 14466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:45133: Network error: Client connection negotiation failed: client connection to 127.1.8.189:45133: connect: Connection refused (error 111)
I20260924 17:46:15.940855 14473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:15.942965 14473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:15.943092 14473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:15.947101 14473 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.189:45133
I20260924 17:46:15.947238 14540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.189:45133 every 8 connection(s)
I20260924 17:46:15.947583 14473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
I20260924 17:46:15.948549 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:15.950196 14541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:15.950762 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14473
I20260924 17:46:15.950857 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-1/wal/instance
I20260924 17:46:15.951087 1058 external_mini_cluster.cc:1428] Setting key aa0a9b5b3c54bef640c101bf3928156b
Sep 24 17:46:15 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.188@KRBTEST.COM; defaulting to no policy
I20260924 17:46:15.963711 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.952678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.1.8.190:39033)
0924 17:46:15.952857 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:15.952860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:15.952872 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:46:15.952902 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:15.952903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:15.952916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:15.952985 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:15.953987 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.954634 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.956267 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.956480 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.956541 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:15.959407 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:15.959409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:15.959409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:15.959418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:15.961576 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:15.962231 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:15.962233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:15.962234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:15.962334 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:15.962776 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:15.962777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:15.962778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:15.962953 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:15.963033 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:15.963472 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 17:46:15.963588 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260924 17:46:15.963935 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.951191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39033 (local address 127.0.0.1:38522)
0924 17:46:15.951428 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.951508 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:15.952251 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:15.953147 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:15.953152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:15.953549 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:15.953864 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:15.953869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.954739 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.954740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:15.956163 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:15.956167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.956592 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.956593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:15.956779 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:15.957364 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:15.957377 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:15.959308 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:15.961703 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:15.961714 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:15.961718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:15.962119 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:15.962469 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:15.962470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:15.962470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:15.962677 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:15.963022 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:15.963030 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:15.963279 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 17:46:15.963422 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 17:46:15.963678 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260924 17:46:15.964812 14541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.188@KRBTEST.COM" created.
Sep 24 17:46:15 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:46:15.973994 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.965150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54862)
0924 17:46:15.965279 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.965291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:15.965339 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:15.965667 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:15.965668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:15.966496 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:15.966572 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:15.966574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.967938 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.967939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:15.968558 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:15.968561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.969083 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.969084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:15.969158 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:15.969653 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:15.969658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:15.971011 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:15.972879 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:15.972880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:15.972881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:15.973055 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:15.973279 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:15.973280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:15.973280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:15.973334 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:15.973646 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:15.973647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:15.973685 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 17:46:15.973751 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:46:15.973872 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":3}
I20260924 17:46:15.974020 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.965304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.1.8.189:45133)
0924 17:46:15.965499 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:15.965505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:15.965527 (+ 22us) server_negotiation.cc:407] Connection header received
0924 17:46:15.965553 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:15.965555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:15.965570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:15.965659 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:15.966658 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.967849 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.968663 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:15.968993 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:15.969086 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:15.971120 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:15.971129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:15.971129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:15.971136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:15.972771 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:15.973151 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:15.973154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:15.973155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:15.973212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:15.973407 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:15.973407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:15.973408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:15.973573 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:15.973671 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:15.973779 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 17:46:15.973921 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":164,"thread_start_us":152,"threads_started":1}
I20260924 17:46:15.974730 14541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:15.975139 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.975004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59744)
0924 17:46:15.975069 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.975118 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:46:15.975289 14492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
I20260924 17:46:15.975756 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:15.975678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59752)
0924 17:46:15.975709 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:15.975736 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 17:46:15.975878 14541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:43091: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab.
Entry for principal kudu/127.1.8.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab.
I20260924 17:46:15.982223 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:43091
--webserver_interface=127.1.8.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab
--principal=kudu/127.1.8.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39033,127.1.8.189:45133,127.1.8.188:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:16.019569 14466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } attempt: 3
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:46:16.025138 14541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } attempt: 1
I20260924 17:46:16.025544 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.025401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59762)
0924 17:46:16.025467 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.025504 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 17:46:16.026077 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.025956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59764)
0924 17:46:16.026008 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.026040 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 17:46:16.026182 14541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:43091: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
I20260924 17:46:16.027678 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.020222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.1.8.189:45133)
0924 17:46:16.020295 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.020297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.020307 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:46:16.020341 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.020342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.020351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.020420 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.020800 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.021336 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.022106 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.022309 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.022390 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.024544 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.024546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.024546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.024584 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.026507 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.026876 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.026877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.026877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.026933 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.027147 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.027148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.027148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.027271 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.027345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.027501 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.027591 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 17:46:16.027683 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.020066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54868)
0924 17:46:16.020154 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.020169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:16.020258 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:16.020425 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:16.020426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:16.020634 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:16.020698 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.020701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.021435 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.021436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.021993 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.021995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.022387 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.022388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.022501 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.023015 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:16.023020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:16.024448 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:16.026609 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.026610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.026611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.026787 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.027008 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.027009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.027010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.027081 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.027337 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:16.027338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:16.027389 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 17:46:16.027477 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:46:16.027587 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 17:46:16.028116 14466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:16.028435 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.028346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59768)
0924 17:46:16.028386 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.028415 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 17:46:16.028749 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.028692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59780)
0924 17:46:16.028715 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.028736 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 17:46:16.028842 14466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:43091: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
I20260924 17:46:16.059994 14466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } attempt: 1
I20260924 17:46:16.060503 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.060365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59788)
0924 17:46:16.060427 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.060470 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:46:16.060911 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.060809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59790)
0924 17:46:16.060862 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.060887 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 17:46:16.061033 14466 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:43091: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
W20260924 17:46:16.070672 14548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:16.070981 14548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:16.071026 14548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:16.072363 14548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:16.072438 14548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:16.072468 14548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:16.072504 14548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:16.072531 14548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:16.074244 14548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39033,127.1.8.189:45133,127.1.8.188:43091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab
--principal=kudu/127.1.8.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.8.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:16.074591 14548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:16.074849 14548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:16.080521 14553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:16.080653 14556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:16.080549 14554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:16.080767 14548 server_base.cc:1034] running on GCE node
I20260924 17:46:16.081036 14548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:16.081292 14548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:16.082468 14548 hybrid_clock.cc:648] HybridClock initialized: now 1790271976082439 us; error 39 us; skew 500 ppm
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:16.085100 14548 init.cc:377] Logged in from keytab as kudu/127.1.8.188@KRBTEST.COM (short username kudu)
I20260924 17:46:16.085661 14548 webserver.cc:460] Webserver started at http://127.1.8.188:40265/ using document root <none> and password file <none>
I20260924 17:46:16.085872 14548 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:16.085915 14548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:16.086007 14548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:16.087158 14548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data/instance:
uuid: "689132865f4a4835af8c2aad0b6c4c1e"
format_stamp: "Formatted at 2026-09-24 17:46:16 on dist-test-slave-r7hf"
server_key: "f189ad0c4e369695bd7dbfc70d7a5b78"
server_key_iv: "492b97d92dd4483e0da3f1507f71e354"
server_key_version: "encryptionkey@0"
I20260924 17:46:16.087491 14548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal/instance:
uuid: "689132865f4a4835af8c2aad0b6c4c1e"
format_stamp: "Formatted at 2026-09-24 17:46:16 on dist-test-slave-r7hf"
server_key: "f189ad0c4e369695bd7dbfc70d7a5b78"
server_key_iv: "492b97d92dd4483e0da3f1507f71e354"
server_key_version: "encryptionkey@0"
I20260924 17:46:16.088781 14548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:46:16.089465 14563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:16.089661 14548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:16.089759 14548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal
uuid: "689132865f4a4835af8c2aad0b6c4c1e"
format_stamp: "Formatted at 2026-09-24 17:46:16 on dist-test-slave-r7hf"
server_key: "f189ad0c4e369695bd7dbfc70d7a5b78"
server_key_iv: "492b97d92dd4483e0da3f1507f71e354"
server_key_version: "encryptionkey@0"
I20260924 17:46:16.089848 14548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:16.094323 14541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } attempt: 2
I20260924 17:46:16.094789 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.094660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59806)
0924 17:46:16.094719 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.094760 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:46:16.095147 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.095054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59812)
0924 17:46:16.095099 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.095123 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 17:46:16.095237 14541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:43091: Network error: Client connection negotiation failed: client connection to 127.1.8.188:43091: connect: Connection refused (error 111)
I20260924 17:46:16.105753 14548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:16.108278 14548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:16.108455 14548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:16.112843 14548 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.188:43091
I20260924 17:46:16.112879 14615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.188:43091 every 8 connection(s)
I20260924 17:46:16.113320 14548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
I20260924 17:46:16.114228 14616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:16.115861 14616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:16.117574 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14548
I20260924 17:46:16.117681 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790271791394214-1058-0/minicluster-data/master-2/wal/instance
I20260924 17:46:16.117888 1058 external_mini_cluster.cc:1428] Setting key dba38726641cbcbf975795ed27507152
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:46:16.126435 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.116603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.1.8.190:39033)
0924 17:46:16.116672 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.116675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.116916 (+ 241us) server_negotiation.cc:407] Connection header received
0924 17:46:16.117385 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.117386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.117396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.117452 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.118338 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.118963 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.120356 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.120648 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.120724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.123347 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.123348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.123349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.123356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.124824 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.125344 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.125345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.125345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.125399 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.125665 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.125666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.125666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.125771 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.125822 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.126186 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.126270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 17:46:16.126526 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.116589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:39033 (local address 127.0.0.1:38538)
0924 17:46:16.116786 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.116826 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:16.117281 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:16.117532 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:16.117538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:16.117925 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:16.118227 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.118234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.119078 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.119079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.120243 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.120245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.120733 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.120733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.120843 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.121422 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:16.121431 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:16.123253 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:16.124909 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.124917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.124919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.125261 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.125469 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.125470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.125471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.125595 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.125839 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:16.125844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:16.126016 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 17:46:16.126135 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 17:46:16.126251 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260924 17:46:16.127409 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.118273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.1.8.188:43091)
0924 17:46:16.118447 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.118452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.118507 (+ 55us) server_negotiation.cc:407] Connection header received
0924 17:46:16.118558 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.118560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.118580 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.118654 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.119198 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.120302 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.121161 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.121561 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.121615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.124159 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.124166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.124167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.124174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.125943 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.126371 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.126374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.126375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.126466 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.126714 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.126715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.126716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.126895 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.127081 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.127199 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.127286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260924 17:46:16.127432 14616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:16.134721 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.127907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:45133 (local address 127.0.0.1:54876)
0924 17:46:16.127963 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.127976 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:16.128019 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:16.128172 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:16.128173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:16.128336 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:16.128402 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.128405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.129327 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.129328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.129976 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.129979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.130464 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.130465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.130535 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.130924 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:16.130930 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:16.132145 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:16.133675 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.133676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.133676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.133832 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.134039 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.134040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.134040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.134108 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.134353 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:16.134355 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:16.134439 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 17:46:16.134509 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:46:16.134622 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 17:46:16.134721 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.127971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.1.8.189:45133)
0924 17:46:16.128024 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.128027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.128038 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:46:16.128087 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.128088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.128097 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.128170 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.128484 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.129255 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.130079 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.130368 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.130460 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.132229 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.132230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.132231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.132238 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.133569 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.133907 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.133908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.133909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.133969 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.134171 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.134171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.134171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.134271 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.134347 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.134543 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.134622 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:46:16.135073 14616 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:46:16.136205 14466 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } attempt: 2
I20260924 17:46:16.137128 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.128340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38542 (local address 127.1.8.190:39033)
0924 17:46:16.128382 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.128383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.128392 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:46:16.128412 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.128413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.128422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.128474 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.128999 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.129675 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.130623 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.130882 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.130950 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.134064 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.134065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.134066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.134073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.135900 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.136266 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.136267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.136268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.136325 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.136548 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.136549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.136549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.136655 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.136875 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.136915 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.136999 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271976, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:46:16.143882 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.135336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59832)
0924 17:46:16.135442 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.135452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:16.135503 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:16.135614 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:16.135615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:16.135764 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:16.135805 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.135807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.136757 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.136758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.137516 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.137517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.137977 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.137978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.138018 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.138459 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:16.138463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:16.139876 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:16.142682 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.142683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.142683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.142890 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.143116 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.143117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.143117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.143175 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.143491 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:16.143493 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:16.143549 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 17:46:16.143683 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 17:46:16.143782 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 17:46:16.144225 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.135399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.1.8.188:43091)
0924 17:46:16.135443 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.135445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.135487 (+ 42us) server_negotiation.cc:407] Connection header received
0924 17:46:16.135555 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.135556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.135564 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.135599 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.135874 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.136679 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.137596 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.137887 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.137937 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.139957 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.139958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.139959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.139964 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.142590 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.142967 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.142968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.142969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.143032 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.143248 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.143249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.143250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.143406 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.143488 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.143705 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.144117 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":333,"server-negotiator.queue_time_us":20}
I20260924 17:46:16.144827 14468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.136475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59848)
0924 17:46:16.136518 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.136528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:16.136576 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:16.136921 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:16.136922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:16.137087 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:16.137126 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.137127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.138160 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.138161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.138993 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.138995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.139583 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.139584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.139639 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.140060 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:16.140064 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:16.141600 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:16.143636 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.143637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.143637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.143858 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.144089 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.144090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.144090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.144155 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.144474 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:16.144475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:16.144528 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 17:46:16.144602 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:46:16.144719 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 17:46:16.145519 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.136638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.1.8.188:43091)
0924 17:46:16.136800 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.136802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.136808 (+ 6us) server_negotiation.cc:407] Connection header received
0924 17:46:16.136823 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.136824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.136833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.137196 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.137328 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.138075 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.139112 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.139444 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.139583 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.141705 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.141706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.141707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.141713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.143556 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.143931 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.143932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.143932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.143988 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.144229 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.144230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.144230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.144384 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.144470 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.145249 (+ 779us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.145359 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260924 17:46:16.145973 14616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: Bootstrap starting.
I20260924 17:46:16.146895 14616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:16.147185 14616 log.cc:824] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:16.147359 14466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: Bootstrap starting.
I20260924 17:46:16.147588 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.137858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.1.8.189:45133)
0924 17:46:16.137897 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.137899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.137906 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:46:16.137939 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.137940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.137948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.137993 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.139703 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.140394 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.141102 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.141365 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.141434 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.144245 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.144246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.144246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.144254 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.146079 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.146518 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.146519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.146520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.146589 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.146820 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.146821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.146822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.146955 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.147142 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.147387 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.147477 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:46:16.148046 14466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:16.148330 14616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: No bootstrap required, opened a new log
I20260924 17:46:16.148341 14466 log.cc:824] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:16.149196 14466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: No bootstrap required, opened a new log
I20260924 17:46:16.149410 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:46:16.150259 14616 raft_consensus.cc:348] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.150462 14616 raft_consensus.cc:374] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:16.150527 14616 raft_consensus.cc:729] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689132865f4a4835af8c2aad0b6c4c1e, State: Initialized, Role: FOLLOWER
I20260924 17:46:16.150707 14616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.151053 14616 sys_catalog.cc:564] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:16.151414 14466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.151575 14466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:16.151625 14466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361b0d0419ce47d4a094d68e37af9b05, State: Initialized, Role: FOLLOWER
I20260924 17:46:16.151778 14466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.152104 14466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:16.152070 14635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.152292 14635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:46:16.152812 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.152935 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:46:16.153312 14641 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:46:16.153364 14641 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:46:16.153404 14548 master_runner.cc:186] Error getting master registration for 127.1.8.190:39033: OK, instance_id { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" instance_seqno: 1790271975792027 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:46:16.156615 14548 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:46:16.154776 14398 master_runner.cc:186] Error getting master registration for 127.1.8.189:45133: OK, instance_id { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" instance_seqno: 1790271975945838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:46:16.157831 14398 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:46:16.161751 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.153416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.1.8.188:43091)
0924 17:46:16.153486 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.153488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.153848 (+ 360us) server_negotiation.cc:407] Connection header received
0924 17:46:16.153947 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.153948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.153961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.153992 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.154794 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.155511 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.156559 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.156880 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.156927 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.158635 (+ 1708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.158637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.158637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.158645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.160297 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.160749 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.160751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.160752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.160814 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.161041 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.161042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.161042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.161203 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.161297 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.161563 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.161654 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 17:46:16.163686 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.150909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.1.8.189:45133)
0924 17:46:16.150956 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.150959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.151187 (+ 228us) server_negotiation.cc:407] Connection header received
0924 17:46:16.151253 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.151254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.151267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.151333 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.152090 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.152841 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.157956 (+ 5115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.158278 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.158322 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.160257 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.160259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.160259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.160266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.162222 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.162659 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.162660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.162660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.162726 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.162962 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.162963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.162964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.163109 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.163220 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.163455 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.163559 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:46:16.164201 14653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:46:16.164322 14653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:46:16.165975 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.153208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.1.8.190:39033)
0924 17:46:16.153250 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.153252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.158029 (+ 4777us) server_negotiation.cc:407] Connection header received
0924 17:46:16.158118 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.158119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.158128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.158156 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.158895 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.159645 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.160675 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.160989 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.161034 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.163149 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.163150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.163151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.163158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.164715 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.165123 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.165124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.165125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.165184 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.165417 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.165418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.165418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.165557 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.165661 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.165806 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.165885 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:46:16.186445 14541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } attempt: 3
Sep 24 17:46:16 dist-test-slave-r7hf krb5kdc[14385](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271975, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:46:16.195483 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.186789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:43091 (local address 127.0.0.1:59858)
0924 17:46:16.186844 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:16.186856 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:16.186895 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:16.187125 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:16.187126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:16.187337 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:16.187391 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.187393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.188338 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.188340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.189201 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:16.189204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.190097 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.190099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:16.190169 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.190726 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:16.190732 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:16.192338 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:16.194293 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.194294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.194294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.194548 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.194772 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:16.194773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:16.194773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:16.194843 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:16.195139 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:16.195140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:16.195190 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 17:46:16.195263 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:46:16.195372 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:46:16.195483 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:16.186890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.1.8.188:43091)
0924 17:46:16.186960 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:16.186962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:16.186971 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:46:16.186998 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:16.186999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:16.187012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:16.187103 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:16.187462 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.188240 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.189698 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:16.190014 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:16.190096 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:16.192416 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:16.192417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:16.192418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:16.192426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:16.194208 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.194636 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.194638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.194638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.194703 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:16.194913 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:16.194914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:16.194914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:16.195069 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:16.195139 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:16.195301 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 17:46:16.195385 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:46:16.197750 14541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: Bootstrap starting.
I20260924 17:46:16.198551 14541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:16.198868 14541 log.cc:824] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:16.199867 14541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: No bootstrap required, opened a new log
I20260924 17:46:16.201911 14541 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.202045 14541 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:16.202090 14541 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea57cf7b45c042ed8fb6dad33097d333, State: Initialized, Role: FOLLOWER
I20260924 17:46:16.202283 14541 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.202641 14658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.202705 14541 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:16.202782 14658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:46:16.204782 14669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:46:16.204900 14669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:46:16.207506 14473 master_runner.cc:263] No leader master found from master ea57cf7b45c042ed8fb6dad33097d333
I20260924 17:46:16.207561 14473 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 17:46:16.289907 14419 tablet.cc:2369] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:46:16.359140 14569 tablet.cc:2369] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:46:16.363721 14635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:16.363838 14635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.364167 14635 leader_election.cc:288] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea57cf7b45c042ed8fb6dad33097d333 (127.1.8.189:45133), 689132865f4a4835af8c2aad0b6c4c1e (127.1.8.188:43091)
I20260924 17:46:16.364496 14590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "361b0d0419ce47d4a094d68e37af9b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689132865f4a4835af8c2aad0b6c4c1e" is_pre_election: true
I20260924 17:46:16.364526 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "361b0d0419ce47d4a094d68e37af9b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea57cf7b45c042ed8fb6dad33097d333" is_pre_election: true
I20260924 17:46:16.364682 14590 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 361b0d0419ce47d4a094d68e37af9b05 in term 0.
I20260924 17:46:16.364681 14515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 361b0d0419ce47d4a094d68e37af9b05 in term 0.
I20260924 17:46:16.364879 14414 leader_election.cc:302] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361b0d0419ce47d4a094d68e37af9b05, ea57cf7b45c042ed8fb6dad33097d333; no voters:
I20260924 17:46:16.365051 14635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:46:16.365106 14635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:46:16.365280 14635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:16.366319 14635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.366510 14635 leader_election.cc:288] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [CANDIDATE]: Term 1 election: Requested vote from peers ea57cf7b45c042ed8fb6dad33097d333 (127.1.8.189:45133), 689132865f4a4835af8c2aad0b6c4c1e (127.1.8.188:43091)
I20260924 17:46:16.366732 14515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "361b0d0419ce47d4a094d68e37af9b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea57cf7b45c042ed8fb6dad33097d333"
I20260924 17:46:16.366847 14515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:16.366811 14590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "361b0d0419ce47d4a094d68e37af9b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689132865f4a4835af8c2aad0b6c4c1e"
I20260924 17:46:16.366940 14590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:16.367667 14515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 361b0d0419ce47d4a094d68e37af9b05 in term 1.
I20260924 17:46:16.367848 14414 leader_election.cc:302] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361b0d0419ce47d4a094d68e37af9b05, ea57cf7b45c042ed8fb6dad33097d333; no voters:
I20260924 17:46:16.367944 14635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:16.368001 14590 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 361b0d0419ce47d4a094d68e37af9b05 in term 1.
I20260924 17:46:16.368075 14635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [term 1 LEADER]: Becoming Leader. State: Replica: 361b0d0419ce47d4a094d68e37af9b05, State: Running, Role: LEADER
I20260924 17:46:16.368180 14635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } }
I20260924 17:46:16.368597 14635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361b0d0419ce47d4a094d68e37af9b05. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.368676 14635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:16.368884 14673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:16.369077 14673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:16.370443 14590 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [term 1 FOLLOWER]: Refusing update from remote peer 361b0d0419ce47d4a094d68e37af9b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:16.370738 14515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [term 1 FOLLOWER]: Refusing update from remote peer 361b0d0419ce47d4a094d68e37af9b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:16.370808 14671 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:16.371003 14635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:16.372097 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361b0d0419ce47d4a094d68e37af9b05. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.372208 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:46:16.372277 14658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361b0d0419ce47d4a094d68e37af9b05. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.372396 14658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:46:16.373183 14671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.373276 14671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:16.373384 14658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.373466 14658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:46:16.373487 14671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.373579 14671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05 [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:16.374058 14675 mvcc.cc:205] Tried to move back new op lower bound from 7332954015210999808 to 7332954015204741120. Current Snapshot: MvccSnapshot[applied={T|T < 7332954015210999808}]
I20260924 17:46:16.374022 14628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "361b0d0419ce47d4a094d68e37af9b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b0d0419ce47d4a094d68e37af9b05" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39033 } } peers { permanent_uuid: "ea57cf7b45c042ed8fb6dad33097d333" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 45133 } } peers { permanent_uuid: "689132865f4a4835af8c2aad0b6c4c1e" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 43091 } } }
I20260924 17:46:16.374150 14628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:46:16.374745 14673 catalog_manager.cc:1347] Generated new cluster ID: c1ce4a3a02114acea7ee8b8620b1f37e
I20260924 17:46:16.374794 14673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:46:16.392220 14673 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:46:16.392877 14673 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:16.400429 14673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 361b0d0419ce47d4a094d68e37af9b05: Generated new TSK 0
I20260924 17:46:16.400838 14673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:46:17.153885 14641 catalog_manager.cc:1259] Loaded cluster ID: c1ce4a3a02114acea7ee8b8620b1f37e
I20260924 17:46:17.153965 14641 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: loading cluster ID for follower catalog manager: success
I20260924 17:46:17.154860 14641 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: acquiring CA information for follower catalog manager: success
I20260924 17:46:17.155180 14641 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 689132865f4a4835af8c2aad0b6c4c1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:46:17.205366 14669 catalog_manager.cc:1259] Loaded cluster ID: c1ce4a3a02114acea7ee8b8620b1f37e
I20260924 17:46:17.205477 14669 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: loading cluster ID for follower catalog manager: success
I20260924 17:46:17.206305 14669 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: acquiring CA information for follower catalog manager: success
I20260924 17:46:17.206670 14669 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea57cf7b45c042ed8fb6dad33097d333: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:46:17.456758 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14398
I20260924 17:46:17.462363 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14473
I20260924 17:46:17.467242 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14548
2026-09-24T17:46:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3039 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:46:17 dist-test-slave-r7hf krb5kdc[14689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:46:17 dist-test-slave-r7hf krb5kdc[14689](info): set up 2 sockets
Sep 24 17:46:17 dist-test-slave-r7hf krb5kdc[14689](info): commencing operation
krb5kdc: starting...
W20260924 17:46:18.520253 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:46:18 dist-test-slave-r7hf krb5kdc[14689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:46:18.649847 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:33523
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:18.734504 14702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:18.734745 14702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:18.734771 14702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:18.735956 14702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:18.735997 14702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:18.736011 14702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:18.736024 14702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:18.736037 14702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:18.737470 14702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:33523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:18.737673 14702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:18.737848 14702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:18.742864 14707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:18.742839 14710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:18.742906 14708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:18.742955 14702 server_base.cc:1034] running on GCE node
I20260924 17:46:18.743366 14702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:18.743669 14702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:18.744837 14702 hybrid_clock.cc:648] HybridClock initialized: now 1790271978744824 us; error 47 us; skew 500 ppm
Sep 24 17:46:18 dist-test-slave-r7hf krb5kdc[14689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271978, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:18.747350 14702 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:18.747803 14702 webserver.cc:460] Webserver started at http://127.1.8.190:35703/ using document root <none> and password file <none>
I20260924 17:46:18.748054 14702 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:18.748126 14702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:18.748275 14702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:18.749375 14702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "a7ef50262c8d4b919f11021f07d5ee3a"
format_stamp: "Formatted at 2026-09-24 17:46:18 on dist-test-slave-r7hf"
server_key: "37bb88702082fe806d38d90a94acaaf1"
server_key_iv: "a990e7eaeeec061596e965cd871454e3"
server_key_version: "encryptionkey@0"
I20260924 17:46:18.749747 14702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "a7ef50262c8d4b919f11021f07d5ee3a"
format_stamp: "Formatted at 2026-09-24 17:46:18 on dist-test-slave-r7hf"
server_key: "37bb88702082fe806d38d90a94acaaf1"
server_key_iv: "a990e7eaeeec061596e965cd871454e3"
server_key_version: "encryptionkey@0"
I20260924 17:46:18.751215 14702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:46:18.752094 14717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:18.752277 14702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:18.752384 14702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "a7ef50262c8d4b919f11021f07d5ee3a"
format_stamp: "Formatted at 2026-09-24 17:46:18 on dist-test-slave-r7hf"
server_key: "37bb88702082fe806d38d90a94acaaf1"
server_key_iv: "a990e7eaeeec061596e965cd871454e3"
server_key_version: "encryptionkey@0"
I20260924 17:46:18.752470 14702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:18.763091 14702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:18.765015 14702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:18.765192 14702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:18.769304 14702 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:33523
I20260924 17:46:18.769340 14769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:33523 every 8 connection(s)
I20260924 17:46:18.769717 14702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:18.770689 14770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:18.773118 14770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a: Bootstrap starting.
I20260924 17:46:18.773736 14770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:18.774030 14770 log.cc:824] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:18.774847 14770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a: No bootstrap required, opened a new log
I20260924 17:46:18.775138 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14702
I20260924 17:46:18.775249 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:46:18.775451 1058 external_mini_cluster.cc:1428] Setting key 1d91a25a0aa8d4aa4712f320be8680db
I20260924 17:46:18.777149 14770 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33523 } }
I20260924 17:46:18.777343 14770 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:18.777383 14770 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ef50262c8d4b919f11021f07d5ee3a, State: Initialized, Role: FOLLOWER
I20260924 17:46:18.777537 14770 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33523 } }
I20260924 17:46:18.777637 14770 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:18.777673 14770 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:18.777721 14770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:18.778281 14770 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33523 } }
I20260924 17:46:18.778369 14770 leader_election.cc:302] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7ef50262c8d4b919f11021f07d5ee3a; no voters:
I20260924 17:46:18.778606 14770 leader_election.cc:288] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:46:18.778626 14775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:18.778741 14775 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [term 1 LEADER]: Becoming Leader. State: Replica: a7ef50262c8d4b919f11021f07d5ee3a, State: Running, Role: LEADER
I20260924 17:46:18.778883 14770 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:18.778884 14775 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33523 } }
I20260924 17:46:18.779353 14777 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33523 } } }
I20260924 17:46:18.779372 14776 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7ef50262c8d4b919f11021f07d5ee3a. Latest consensus state: current_term: 1 leader_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7ef50262c8d4b919f11021f07d5ee3a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 33523 } } }
I20260924 17:46:18.779521 14777 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:18.779539 14776 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a [sys.catalog]: This master's current role is: LEADER
W20260924 17:46:18.780874 14791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:46:18.780977 14791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:46:18.781067 14783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:18.781270 14783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:18.783164 14783 catalog_manager.cc:1347] Generated new cluster ID: 54c1c294feb24e6e83f8c31a9a22e471
I20260924 17:46:18.783217 14783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:46:18 dist-test-slave-r7hf krb5kdc[14689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:18.787621 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:18.775971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.1.8.190:33523)
0924 17:46:18.776175 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:18.776183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:18.776209 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:46:18.776682 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:18.776688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:18.776736 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:18.776852 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:18.777677 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:18.779194 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:18.780158 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:18.780765 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:18.780871 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:18.783739 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:18.783743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:18.783743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:18.783755 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:18.785771 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:18.786156 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:18.786160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:18.786160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:18.786273 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:18.786484 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:18.786485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:18.786485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:18.786808 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:18.787004 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:18.787105 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 17:46:18.787409 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260924 17:46:18.788877 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:46:18.795918 14783 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:46:18.796636 14783 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:18.804399 14783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7ef50262c8d4b919f11021f07d5ee3a: Generated new TSK 0
I20260924 17:46:18.804610 14783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:46:18.843164 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14702
2026-09-24T17:46:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1374 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:46:18 dist-test-slave-r7hf krb5kdc[14797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:46:18 dist-test-slave-r7hf krb5kdc[14797](info): set up 2 sockets
Sep 24 17:46:18 dist-test-slave-r7hf krb5kdc[14797](info): commencing operation
krb5kdc: starting...
W20260924 17:46:19.878558 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:46:19 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:46:20.006053 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37595
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:20.089825 14810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:20.090128 14810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:20.090171 14810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:20.091480 14810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:20.091574 14810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:20.091600 14810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:20.091620 14810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:20.091648 14810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:20.093839 14810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:37595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:20.094172 14810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:20.094440 14810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:46:20.099809 14810 server_base.cc:1034] running on GCE node
W20260924 17:46:20.099895 14816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.099822 14815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.099961 14818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.100461 14810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:20.100796 14810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:20.101969 14810 hybrid_clock.cc:648] HybridClock initialized: now 1790271980101948 us; error 53 us; skew 500 ppm
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:20.113288 14810 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:20.113924 14810 webserver.cc:460] Webserver started at http://127.1.8.190:45243/ using document root <none> and password file <none>
I20260924 17:46:20.114189 14810 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:20.114266 14810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:20.114496 14810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:20.115657 14810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "68843f529c1f4f15addf79eb7f10b9cd"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "020d5050e60b2d59c3de101b9076a6f3"
server_key_iv: "2bcd045ff84b8065a5978c3562a288cb"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.116060 14810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "68843f529c1f4f15addf79eb7f10b9cd"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "020d5050e60b2d59c3de101b9076a6f3"
server_key_iv: "2bcd045ff84b8065a5978c3562a288cb"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.117465 14810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:46:20.118294 14825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.118511 14810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:20.118630 14810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "68843f529c1f4f15addf79eb7f10b9cd"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "020d5050e60b2d59c3de101b9076a6f3"
server_key_iv: "2bcd045ff84b8065a5978c3562a288cb"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.118721 14810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:20.137267 14810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:20.139863 14810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:20.140048 14810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:20.144577 14810 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:37595
I20260924 17:46:20.144613 14877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:37595 every 8 connection(s)
I20260924 17:46:20.144997 14810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:20.146023 14878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:20.148455 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap starting.
I20260924 17:46:20.149089 14878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:20.149392 14878 log.cc:824] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:20.150240 14878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: No bootstrap required, opened a new log
I20260924 17:46:20.151396 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14810
I20260924 17:46:20.151525 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:46:20.151732 1058 external_mini_cluster.cc:1428] Setting key 28277a7acc210773e9f43a31ba5c8cd9
I20260924 17:46:20.152442 14878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.152647 14878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:20.152678 14878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68843f529c1f4f15addf79eb7f10b9cd, State: Initialized, Role: FOLLOWER
I20260924 17:46:20.152853 14878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.152925 14878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:20.153039 14878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:20.153110 14878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:20.153997 14878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.154100 14878 leader_election.cc:302] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68843f529c1f4f15addf79eb7f10b9cd; no voters:
I20260924 17:46:20.154333 14878 leader_election.cc:288] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:46:20.154415 14883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:20.154572 14883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 LEADER]: Becoming Leader. State: Replica: 68843f529c1f4f15addf79eb7f10b9cd, State: Running, Role: LEADER
I20260924 17:46:20.154642 14878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:20.154716 14883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.155093 14883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68843f529c1f4f15addf79eb7f10b9cd. Latest consensus state: current_term: 1 leader_uuid: "68843f529c1f4f15addf79eb7f10b9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } } }
I20260924 17:46:20.155315 14883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:20.155222 14885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68843f529c1f4f15addf79eb7f10b9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } } }
I20260924 17:46:20.155591 14892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:20.155599 14885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:20.156181 14892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:20.160151 14892 catalog_manager.cc:1347] Generated new cluster ID: f441ac147c514b2f8e77554bca7c5ab7
I20260924 17:46:20.160214 14892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:46:20.164034 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.152359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.1.8.190:37595)
0924 17:46:20.152580 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.152589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.152617 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:46:20.152759 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.152767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.152826 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.153002 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:20.153877 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.155228 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.156484 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.157112 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.157232 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.160178 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:20.160183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:20.160183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:20.160195 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:20.162170 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.162721 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.162725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.162726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.162860 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.163103 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.163104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.163105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.163386 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:20.163579 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.163665 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.163847 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260924 17:46:20.170769 14892 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:46:20.171399 14892 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:20.177047 14892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Generated new TSK 0
I20260924 17:46:20.177331 14892 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:46:20.209975 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37595
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:20.317381 14904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:20.317662 14904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:20.317710 14904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:20.319219 14904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:20.319303 14904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:20.319356 14904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:20.321106 14904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:20.321485 14904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:20.321753 14904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:46:20.327695 14904 server_base.cc:1034] running on GCE node
W20260924 17:46:20.327621 14912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.327652 14909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.327628 14910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.328209 14904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:20.328550 14904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:20.329752 14904 hybrid_clock.cc:648] HybridClock initialized: now 1790271980329728 us; error 39 us; skew 500 ppm
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:20.332667 14904 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:20.333256 14904 webserver.cc:460] Webserver started at http://127.1.8.129:33425/ using document root <none> and password file <none>
I20260924 17:46:20.333549 14904 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:20.333630 14904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:20.333796 14904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:20.335114 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "ac8a5f8b95db50d93752690e5172e569"
server_key_iv: "e1d27cb7c036974f4acdcf1450e1ad9c"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.335531 14904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "ac8a5f8b95db50d93752690e5172e569"
server_key_iv: "e1d27cb7c036974f4acdcf1450e1ad9c"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.337265 14904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:46:20.338256 14919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.338479 14904 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:20.338583 14904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "ac8a5f8b95db50d93752690e5172e569"
server_key_iv: "e1d27cb7c036974f4acdcf1450e1ad9c"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.338675 14904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:20.362211 14904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:20.364442 14904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:20.364656 14904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:20.364924 14904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:20.365275 14904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:20.365334 14904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.365377 14904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:20.365403 14904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.371263 14904 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:33843
I20260924 17:46:20.371347 15032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:33843 every 8 connection(s)
I20260924 17:46:20.371695 14904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:20.376467 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 14904
I20260924 17:46:20.376603 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:46:20.376827 1058 external_mini_cluster.cc:1428] Setting key 86a075a1bff17af31d7843247b58cf43
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:20.382186 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.372566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:52187 (local address 127.1.8.190:37595)
0924 17:46:20.372625 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.372628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.372865 (+ 237us) server_negotiation.cc:407] Connection header received
0924 17:46:20.373346 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.373347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.373361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.373415 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:20.374370 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.375013 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.376027 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.376218 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.376272 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.379173 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:20.379175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:20.379175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:20.379185 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:20.380656 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.381196 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.381197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.381197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.381246 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.381531 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.381531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.381532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.381634 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:20.381685 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.382051 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.382120 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:46:20.382582 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.372545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:52187)
0924 17:46:20.372724 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.372765 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.373256 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.373471 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.373474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.374077 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:20.374279 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.374284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.375105 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.375106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.375944 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.375946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.376324 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.376325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.376483 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.377137 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:20.377148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:20.379065 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:20.380748 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:20.380756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:20.380758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:20.381119 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:20.381313 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:20.381314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:20.381314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:20.381468 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:20.381703 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:20.381710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:20.381895 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.382035 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.382228 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260924 17:46:20.383152 15033 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37595
I20260924 17:46:20.383270 15033 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:20.383499 15033 heartbeater.cc:511] Master 127.1.8.190:37595 requested a full tablet report, sending...
I20260924 17:46:20.384057 14842 ts_manager.cc:194] Registered new tserver with Master: 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843)
I20260924 17:46:20.384693 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:52187
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:46:20.406580 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37595
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:20.489898 15038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:20.490175 15038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:20.490214 15038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:20.491449 15038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:20.491546 15038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:20.491593 15038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:20.493155 15038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:20.493443 15038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:20.493696 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:20.498989 15046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.499185 15038 server_base.cc:1034] running on GCE node
W20260924 17:46:20.499068 15043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.499285 15044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.499527 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:20.499749 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:20.500902 15038 hybrid_clock.cc:648] HybridClock initialized: now 1790271980500880 us; error 48 us; skew 500 ppm
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:20.503628 15038 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:20.504159 15038 webserver.cc:460] Webserver started at http://127.1.8.130:43503/ using document root <none> and password file <none>
I20260924 17:46:20.504396 15038 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:20.504465 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:20.504621 15038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:20.505739 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "b901152d553af4f0ac93e01973a414d8"
server_key_iv: "c24a34e79c1ab6b8cea497c56ac641c8"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.506125 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "b901152d553af4f0ac93e01973a414d8"
server_key_iv: "c24a34e79c1ab6b8cea497c56ac641c8"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.507642 15038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:46:20.508415 15053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.508594 15038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:20.508702 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "b901152d553af4f0ac93e01973a414d8"
server_key_iv: "c24a34e79c1ab6b8cea497c56ac641c8"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.508780 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:20.520035 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:20.522591 15038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:20.522761 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:20.523015 15038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:20.523325 15038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:20.523378 15038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.523414 15038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:20.523445 15038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.529129 15038 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39597
I20260924 17:46:20.529224 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39597 every 8 connection(s)
I20260924 17:46:20.529564 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:46:20.531713 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15038
I20260924 17:46:20.531829 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:46:20.532037 1058 external_mini_cluster.cc:1428] Setting key 932b3f077f10deda86b9ca33598e3ef2
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:20.540051 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.530321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:38741 (local address 127.1.8.190:37595)
0924 17:46:20.530382 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.530385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.530639 (+ 254us) server_negotiation.cc:407] Connection header received
0924 17:46:20.531136 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.531137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.531146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.531208 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:20.532316 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.533011 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.534105 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.534312 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.534373 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.537004 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:20.537006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:20.537006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:20.537015 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:20.538574 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.539014 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.539015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.539016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.539082 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.539346 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.539347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.539348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.539474 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:20.539569 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.539858 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.539952 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
I20260924 17:46:20.540330 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.530295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:38741)
0924 17:46:20.530506 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.530549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.531053 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.531263 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.531267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.531929 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:20.532215 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.532221 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.533118 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.533119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.533986 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.533990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.534442 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.534443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.534605 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.535165 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:20.535176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:20.536885 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:20.538673 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:20.538681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:20.538684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:20.538931 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:20.539154 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:20.539155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:20.539155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:20.539276 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:20.539558 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:20.539565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:20.539722 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.539837 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.540023 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":132,"threads_started":1}
I20260924 17:46:20.540897 15167 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37595
I20260924 17:46:20.541035 15167 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:20.541258 15167 heartbeater.cc:511] Master 127.1.8.190:37595 requested a full tablet report, sending...
I20260924 17:46:20.541743 14842 ts_manager.cc:194] Registered new tserver with Master: ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597)
I20260924 17:46:20.542112 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:38741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:46:20.557647 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37595
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:20.640100 15172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:20.640336 15172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:20.640363 15172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:20.641803 15172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:20.641885 15172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:20.641932 15172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:20.643550 15172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:20.643762 15172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:20.643958 15172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:20.649400 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.649437 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.649400 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.649645 15172 server_base.cc:1034] running on GCE node
I20260924 17:46:20.649855 15172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:20.650094 15172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:20.651235 15172 hybrid_clock.cc:648] HybridClock initialized: now 1790271980651221 us; error 30 us; skew 500 ppm
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:20.653900 15172 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:20.654572 15172 webserver.cc:460] Webserver started at http://127.1.8.131:42493/ using document root <none> and password file <none>
I20260924 17:46:20.654815 15172 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:20.654863 15172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:20.655049 15172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:20.656229 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "e67af083c1f3421182e9017026a62b8f"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "c38197db996ae7cb2bad0d8b83ccae96"
server_key_iv: "071eab1c9900c154369ce6d12f7618ac"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.656693 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "e67af083c1f3421182e9017026a62b8f"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "c38197db996ae7cb2bad0d8b83ccae96"
server_key_iv: "071eab1c9900c154369ce6d12f7618ac"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.658041 15172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:46:20.658814 15187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.659034 15172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:20.659147 15172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "e67af083c1f3421182e9017026a62b8f"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "c38197db996ae7cb2bad0d8b83ccae96"
server_key_iv: "071eab1c9900c154369ce6d12f7618ac"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.659226 15172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:20.690294 15172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:20.692854 15172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:20.693042 15172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:20.693298 15172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:20.693632 15172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:20.693688 15172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.693732 15172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:20.693758 15172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.699988 15172 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:38035
I20260924 17:46:20.700027 15300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:38035 every 8 connection(s)
I20260924 17:46:20.700366 15172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:46:20.703086 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15172
I20260924 17:46:20.703181 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:46:20.703354 1058 external_mini_cluster.cc:1428] Setting key e9abbdf1b340cde1018727a1a9e684bc
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:20.709993 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.701100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:44051 (local address 127.1.8.190:37595)
0924 17:46:20.701146 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.701148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.701344 (+ 196us) server_negotiation.cc:407] Connection header received
0924 17:46:20.701911 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.701912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.701925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.701985 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:20.703121 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.703833 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.704886 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.705086 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.705142 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.707305 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:20.707307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:20.707307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:20.707314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:20.708776 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.709169 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.709170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.709170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.709224 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.709455 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.709455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.709456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.709563 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:20.709613 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.709861 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.709927 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:46:20.710256 15303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.701060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:44051)
0924 17:46:20.701218 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.701255 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.701796 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.702063 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.702067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.702738 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:20.703006 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.703012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.703920 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.703921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.704809 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.704812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.705162 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.705163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.705261 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.705742 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:20.705751 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:20.707211 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:20.708856 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:20.708862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:20.708864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:20.709103 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:20.709289 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:20.709290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:20.709290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:20.709399 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:20.709620 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:20.709624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:20.709749 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.709831 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.709959 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260924 17:46:20.710805 15301 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37595
I20260924 17:46:20.710903 15301 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:20.711058 15301 heartbeater.cc:511] Master 127.1.8.190:37595 requested a full tablet report, sending...
I20260924 17:46:20.711377 14842 ts_manager.cc:194] Registered new tserver with Master: e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
I20260924 17:46:20.711818 14842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:44051
I20260924 17:46:20.715193 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:46:20.722674 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.716487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.1.8.190:37595)
0924 17:46:20.716581 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.716583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.716680 (+ 97us) server_negotiation.cc:407] Connection header received
0924 17:46:20.716749 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.716750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.716761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.716819 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:20.717389 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.717880 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.718732 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.718934 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.719018 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.720205 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:20.720206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:20.720207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:20.720214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:20.721395 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.721754 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.721755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.721755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.721814 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:20.722042 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:20.722043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:20.722043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:20.722205 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:20.722312 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.722462 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.722567 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 17:46:20.724279 14842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:46:20.725828 14842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:46:20.732682 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.728299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33843 (local address 127.0.0.1:44508)
0924 17:46:20.728470 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.728857 (+ 387us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.728901 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.729163 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.729164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.729166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:20.729227 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.729229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.730115 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.730117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.731424 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.731427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.732347 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.732348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.732444 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.732463 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.732536 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.732593 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":48,"threads_started":1}
I20260924 17:46:20.732877 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.728582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.1.8.130:39597)
0924 17:46:20.728890 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.728893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.728902 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:46:20.728937 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.728939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.728955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.729022 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:20.729289 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.730441 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.731676 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.732353 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.732416 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.732463 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.732737 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.732785 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":83,"threads_started":1}
I20260924 17:46:20.732931 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.728313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39597 (local address 127.0.0.1:47940)
0924 17:46:20.728690 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.728700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.728771 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.729114 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.729117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.729122 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:20.729204 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.729206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.730558 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.730559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.731599 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.731601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.732503 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.732504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.732591 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.732596 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.732647 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.732691 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260924 17:46:20.732703 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.728468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44508 (local address 127.1.8.129:33843)
0924 17:46:20.728684 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.728690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.728903 (+ 213us) server_negotiation.cc:407] Connection header received
0924 17:46:20.728987 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.728990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.729007 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.729078 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:20.729307 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.730011 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.731506 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.732242 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.732323 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.732391 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.732567 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.732615 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260924 17:46:20.733728 15101 tablet_service.cc:1467] Processing CreateTablet for tablet a7999e8ddede49e68fe545585aab5d2d (DEFAULT_TABLE table=test-table [id=4c2d8981ae024fada9865d2ca61ade58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:20.733798 14967 tablet_service.cc:1467] Processing CreateTablet for tablet a7999e8ddede49e68fe545585aab5d2d (DEFAULT_TABLE table=test-table [id=4c2d8981ae024fada9865d2ca61ade58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:20.734094 14967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7999e8ddede49e68fe545585aab5d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:20.734103 15101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7999e8ddede49e68fe545585aab5d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:20.734891 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.728311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:38035 (local address 127.0.0.1:42410)
0924 17:46:20.728655 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.728700 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.728771 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.729114 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.729117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.729120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:20.729203 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.729206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.731667 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.731668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.732378 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.732380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.734643 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.734644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.734712 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.734718 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.734776 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.734819 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":8,"thread_start_us":100,"threads_started":1}
I20260924 17:46:20.734921 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.728542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.1.8.131:38035)
0924 17:46:20.728788 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.728792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.728801 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:46:20.728896 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.728898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.728918 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.729021 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:20.729290 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.730446 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.732476 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.734553 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.734630 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.734687 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.734798 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.734842 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260924 17:46:20.735913 15235 tablet_service.cc:1467] Processing CreateTablet for tablet a7999e8ddede49e68fe545585aab5d2d (DEFAULT_TABLE table=test-table [id=4c2d8981ae024fada9865d2ca61ade58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:20.736270 15235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7999e8ddede49e68fe545585aab5d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:20.737011 15318 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap starting.
I20260924 17:46:20.737551 15319 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Bootstrap starting.
I20260924 17:46:20.737692 15318 tablet_bootstrap.cc:654] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:20.738245 15319 tablet_bootstrap.cc:654] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:20.738263 15318 log.cc:824] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:20.738816 15319 log.cc:824] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:20.739145 15318 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: No bootstrap required, opened a new log
I20260924 17:46:20.739192 15318 ts_tablet_manager.cc:1397] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:20.739573 15319 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: No bootstrap required, opened a new log
I20260924 17:46:20.739645 15319 ts_tablet_manager.cc:1397] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:46:20.741451 15318 raft_consensus.cc:348] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.741652 15318 raft_consensus.cc:374] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:20.741689 15318 raft_consensus.cc:729] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79aba118bf9f464b9c4cfd5d7e1c1ae3, State: Initialized, Role: FOLLOWER
I20260924 17:46:20.741607 15319 raft_consensus.cc:348] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.741771 15319 raft_consensus.cc:374] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:20.741813 15319 raft_consensus.cc:729] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1ad90a2a484856a5f113f4c7930a7b, State: Initialized, Role: FOLLOWER
I20260924 17:46:20.741824 15318 consensus_queue.cc:260] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.741926 15319 consensus_queue.cc:260] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.742141 15318 ts_tablet_manager.cc:1428] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:46:20.742187 15319 ts_tablet_manager.cc:1428] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 17:46:20.742254 15167 heartbeater.cc:503] Master 127.1.8.190:37595 was elected leader, sending a full tablet report...
I20260924 17:46:20.742268 15033 heartbeater.cc:503] Master 127.1.8.190:37595 was elected leader, sending a full tablet report...
I20260924 17:46:20.742373 15322 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Bootstrap starting.
I20260924 17:46:20.743142 15322 tablet_bootstrap.cc:654] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:20.743682 15322 log.cc:824] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:20.744369 15322 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: No bootstrap required, opened a new log
I20260924 17:46:20.744424 15322 ts_tablet_manager.cc:1397] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:46:20.745813 15322 raft_consensus.cc:348] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.745955 15322 raft_consensus.cc:374] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:20.745978 15322 raft_consensus.cc:729] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67af083c1f3421182e9017026a62b8f, State: Initialized, Role: FOLLOWER
I20260924 17:46:20.746133 15322 consensus_queue.cc:260] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.746481 15322 ts_tablet_manager.cc:1428] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:46:20.746481 15301 heartbeater.cc:503] Master 127.1.8.190:37595 was elected leader, sending a full tablet report...
I20260924 17:46:20.775174 15323 raft_consensus.cc:482] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:20.775298 15323 raft_consensus.cc:504] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.775772 15323 leader_election.cc:288] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
I20260924 17:46:20.779168 15317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.776082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:57337 (local address 127.1.8.130:39597)
0924 17:46:20.776143 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.776145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.776383 (+ 238us) server_negotiation.cc:407] Connection header received
0924 17:46:20.776445 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.776446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.776459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.776508 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:20.776648 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.777198 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.778187 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.778770 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.778832 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.778842 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.779042 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.779089 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:46:20.779224 15035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.775945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:38035 (local address 127.1.8.129:36189)
0924 17:46:20.776007 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.776020 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.776069 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.776246 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.776247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.776248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:20.776327 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.776329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.777178 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.777179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.778024 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.778027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.778837 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.778839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.778962 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.778966 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.779029 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.779086 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 17:46:20.779201 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.776010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:36189 (local address 127.1.8.131:38035)
0924 17:46:20.776062 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.776064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.776074 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:46:20.776157 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.776158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.776178 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.776241 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:20.776433 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.777111 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.778124 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.778735 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.778808 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.778820 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.779035 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.779105 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:46:20.779425 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.775965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39597 (local address 127.1.8.129:57337)
0924 17:46:20.776289 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.776299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:20.776363 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:20.776512 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:20.776513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:20.776514 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:20.776579 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.776580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.777276 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.777278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.778062 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:20.778064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.778867 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.778868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:20.778949 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.778954 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:46:20.779267 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 17:46:20.779320 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":120,"thread_start_us":132,"threads_started":1}
I20260924 17:46:20.779729 15121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7999e8ddede49e68fe545585aab5d2d" candidate_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" is_pre_election: true
I20260924 17:46:20.779736 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7999e8ddede49e68fe545585aab5d2d" candidate_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67af083c1f3421182e9017026a62b8f" is_pre_election: true
I20260924 17:46:20.779907 15121 raft_consensus.cc:2454] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79aba118bf9f464b9c4cfd5d7e1c1ae3 in term 0.
I20260924 17:46:20.779925 15255 raft_consensus.cc:2454] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79aba118bf9f464b9c4cfd5d7e1c1ae3 in term 0.
W20260924 17:46:20.780086 15168 tablet.cc:2369] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:46:20.780172 14920 leader_election.cc:302] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3, ed1ad90a2a484856a5f113f4c7930a7b; no voters:
I20260924 17:46:20.780349 15323 raft_consensus.cc:2781] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:46:20.780438 15323 raft_consensus.cc:482] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:46:20.780473 15323 raft_consensus.cc:3037] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:20.781213 15323 raft_consensus.cc:504] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.781381 15323 leader_election.cc:288] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 1 election: Requested vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
I20260924 17:46:20.781572 15255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7999e8ddede49e68fe545585aab5d2d" candidate_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67af083c1f3421182e9017026a62b8f"
I20260924 17:46:20.781651 15255 raft_consensus.cc:3037] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:20.782384 15255 raft_consensus.cc:2454] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79aba118bf9f464b9c4cfd5d7e1c1ae3 in term 1.
I20260924 17:46:20.782545 15121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7999e8ddede49e68fe545585aab5d2d" candidate_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
I20260924 17:46:20.782640 14923 leader_election.cc:302] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3, e67af083c1f3421182e9017026a62b8f; no voters:
I20260924 17:46:20.782660 15121 raft_consensus.cc:3037] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:20.782732 15323 raft_consensus.cc:2781] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:20.782915 15323 raft_consensus.cc:686] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 79aba118bf9f464b9c4cfd5d7e1c1ae3, State: Running, Role: LEADER
I20260924 17:46:20.783041 15323 consensus_queue.cc:237] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:20.783630 15121 raft_consensus.cc:2454] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79aba118bf9f464b9c4cfd5d7e1c1ae3 in term 1.
I20260924 17:46:20.783946 14842 catalog_manager.cc:5495] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } health_report { overall_health: UNKNOWN } } }
I20260924 17:46:20.793947 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.791220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.1.8.129:33843)
0924 17:46:20.791277 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:20.791281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:20.791294 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:46:20.791326 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:20.791327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:20.791345 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:20.791403 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:20.791525 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.791978 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.792961 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:20.793326 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:20.793417 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:20.793586 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:20.793686 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:20.793751 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:20.793793 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 17:46:20.793813 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 17:46:20.795781 15255 raft_consensus.cc:1261] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Refusing update from remote peer 79aba118bf9f464b9c4cfd5d7e1c1ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:20.795781 15121 raft_consensus.cc:1261] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Refusing update from remote peer 79aba118bf9f464b9c4cfd5d7e1c1ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:20.796152 15328 consensus_queue.cc:1035] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:20.796303 15323 consensus_queue.cc:1035] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:20.799588 15332 mvcc.cc:205] Tried to move back new op lower bound from 7332954033336811520 to 7332954033287942144. Current Snapshot: MvccSnapshot[applied={T|T < 7332954033336811520}]
I20260924 17:46:20.799715 15333 mvcc.cc:205] Tried to move back new op lower bound from 7332954033336811520 to 7332954033287942144. Current Snapshot: MvccSnapshot[applied={T|T < 7332954033336811520 or (T in {7332954033336811520})}]
I20260924 17:46:20.800997 14841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:20.801121 14841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:20.801940 14841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Sending DeleteTablet for 3 replicas of tablet a7999e8ddede49e68fe545585aab5d2d
I20260924 17:46:20.802311 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14904
I20260924 17:46:20.802311 15235 tablet_service.cc:1514] Processing DeleteTablet for tablet a7999e8ddede49e68fe545585aab5d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:20 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42410
I20260924 17:46:20.802313 15101 tablet_service.cc:1514] Processing DeleteTablet for tablet a7999e8ddede49e68fe545585aab5d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:20 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:47940
I20260924 17:46:20.803926 15341 tablet_replica.cc:330] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: stopping tablet replica
I20260924 17:46:20.804046 15341 raft_consensus.cc:2229] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:20.804164 15341 raft_consensus.cc:2258] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:20.804368 15340 tablet_replica.cc:330] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: stopping tablet replica
I20260924 17:46:20.804484 15340 raft_consensus.cc:2229] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:20.804595 15340 raft_consensus.cc:2258] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:20.804741 15341 ts_tablet_manager.cc:1905] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:20.805015 15340 ts_tablet_manager.cc:1905] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:20.806756 15340 ts_tablet_manager.cc:1918] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:20.806824 15341 ts_tablet_manager.cc:1918] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:20.806861 15340 log.cc:1196] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/a7999e8ddede49e68fe545585aab5d2d
I20260924 17:46:20.806892 15341 log.cc:1196] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/a7999e8ddede49e68fe545585aab5d2d
I20260924 17:46:20.807168 15340 ts_tablet_manager.cc:1939] T a7999e8ddede49e68fe545585aab5d2d P ed1ad90a2a484856a5f113f4c7930a7b: Deleting consensus metadata
I20260924 17:46:20.807188 15341 ts_tablet_manager.cc:1939] T a7999e8ddede49e68fe545585aab5d2d P e67af083c1f3421182e9017026a62b8f: Deleting consensus metadata
I20260924 17:46:20.807641 14826 catalog_manager.cc:4841] TS ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597): tablet a7999e8ddede49e68fe545585aab5d2d (table test-table [id=4c2d8981ae024fada9865d2ca61ade58]) successfully deleted
I20260924 17:46:20.807674 14829 catalog_manager.cc:4841] TS e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): tablet a7999e8ddede49e68fe545585aab5d2d (table test-table [id=4c2d8981ae024fada9865d2ca61ade58]) successfully deleted
W20260924 17:46:20.808202 14828 connection.cc:535] client connection to 127.1.8.129:33843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 17:46:20.808420 14828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:46:20.808449 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15038
I20260924 17:46:20.809562 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:20.809419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33843 (local address 127.0.0.1:44534)
0924 17:46:20.809479 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:20.809521 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 17:46:20.809700 14828 catalog_manager.cc:4584] TS 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7999e8ddede49e68fe545585aab5d2d: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33843: connect: Connection refused (error 111)
I20260924 17:46:20.813822 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15172
I20260924 17:46:20.819044 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 14810
I20260924 17:46:20.824635 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37595
--webserver_interface=127.1.8.190
--webserver_port=45243
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:20.907225 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:20.907521 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:20.907562 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:20.908784 15343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:20.908867 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:20.908891 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:20.908913 15343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:20.908944 15343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:20.910573 15343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:37595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:20.910885 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:20.911131 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:20.920320 15349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:20.920354 15350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.920419 15343 server_base.cc:1034] running on GCE node
W20260924 17:46:20.920320 15352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:20.920818 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:20.921116 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:20.922322 15343 hybrid_clock.cc:648] HybridClock initialized: now 1790271980922309 us; error 27 us; skew 500 ppm
Sep 24 17:46:20 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271980, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:20.925035 15343 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:20.925530 15343 webserver.cc:460] Webserver started at http://127.1.8.190:45243/ using document root <none> and password file <none>
I20260924 17:46:20.925794 15343 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:20.925873 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:20.927404 15343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:20.928062 15359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:20.928264 15343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:20.928391 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "68843f529c1f4f15addf79eb7f10b9cd"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "020d5050e60b2d59c3de101b9076a6f3"
server_key_iv: "2bcd045ff84b8065a5978c3562a288cb"
server_key_version: "encryptionkey@0"
I20260924 17:46:20.928900 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:20.950306 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:20.952278 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:20.952458 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:20.956835 15343 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:37595
I20260924 17:46:20.956869 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:37595 every 8 connection(s)
I20260924 17:46:20.957288 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:20.959057 15412 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:46:20.959668 15412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap starting.
I20260924 17:46:20.959923 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15343
I20260924 17:46:20.960245 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:33843
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:37595
--webserver_port=33425
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:20.968891 15412 log.cc:824] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:20.971921 15412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:20.972316 15412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap complete.
I20260924 17:46:20.976265 15412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.976658 15412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68843f529c1f4f15addf79eb7f10b9cd, State: Initialized, Role: FOLLOWER
I20260924 17:46:20.976847 15412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.976928 15412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:20.976989 15412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:20.977046 15412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:46:20.978271 15412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.978361 15412 leader_election.cc:302] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68843f529c1f4f15addf79eb7f10b9cd; no voters:
I20260924 17:46:20.978564 15412 leader_election.cc:288] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:46:20.978616 15416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:46:20.978726 15416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 LEADER]: Becoming Leader. State: Replica: 68843f529c1f4f15addf79eb7f10b9cd, State: Running, Role: LEADER
I20260924 17:46:20.978839 15416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:20.978869 15412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:20.979135 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68843f529c1f4f15addf79eb7f10b9cd. Latest consensus state: current_term: 2 leader_uuid: "68843f529c1f4f15addf79eb7f10b9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } } }
I20260924 17:46:20.979215 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:20.979143 15417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "68843f529c1f4f15addf79eb7f10b9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } } }
I20260924 17:46:20.979396 15417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:20.979416 15420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:20.980370 15420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:20.980517 15420 catalog_manager.cc:1259] Loaded cluster ID: f441ac147c514b2f8e77554bca7c5ab7
I20260924 17:46:20.980748 15420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:46:20.981019 15420 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 17:46:20.983815 15420 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 17:46:20.987412 15420 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:20.987643 15420 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Loaded TSK: 0
I20260924 17:46:20.987869 15420 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:46:21.055610 15414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:21.055816 15414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:21.055842 15414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:21.057050 15414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:21.057093 15414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:21.057122 15414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:21.058682 15414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:33843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=33425
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:21.058928 15414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:21.059176 15414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:21.064404 15439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.064631 15442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.064728 15440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.065233 15414 server_base.cc:1034] running on GCE node
I20260924 17:46:21.065443 15414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:21.065722 15414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:21.066876 15414 hybrid_clock.cc:648] HybridClock initialized: now 1790271981066860 us; error 33 us; skew 500 ppm
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:21.069541 15414 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:21.070094 15414 webserver.cc:460] Webserver started at http://127.1.8.129:33425/ using document root <none> and password file <none>
I20260924 17:46:21.070345 15414 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:21.070446 15414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:21.071959 15414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:21.072691 15449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.072888 15414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:21.073000 15414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "ac8a5f8b95db50d93752690e5172e569"
server_key_iv: "e1d27cb7c036974f4acdcf1450e1ad9c"
server_key_version: "encryptionkey@0"
I20260924 17:46:21.073480 15414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:21.093226 15414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:21.096014 15414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:21.096180 15414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:21.096469 15414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:21.097013 15456 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:46:21.098657 15414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:46:21.098719 15414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:46:21.098753 15414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:46:21.099619 15414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:46:21.099668 15414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:21.099763 15456 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap starting.
I20260924 17:46:21.107144 15414 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:33843
I20260924 17:46:21.107540 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:33843 every 8 connection(s)
I20260924 17:46:21.107650 15414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:21.111578 15456 log.cc:824] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:21.114329 15456 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:21.114671 15456 tablet_bootstrap.cc:492] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap complete.
I20260924 17:46:21.115878 15456 ts_tablet_manager.cc:1397] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:21.117173 15456 raft_consensus.cc:348] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:21.117483 15456 raft_consensus.cc:729] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79aba118bf9f464b9c4cfd5d7e1c1ae3, State: Initialized, Role: FOLLOWER
I20260924 17:46:21.117600 15456 consensus_queue.cc:260] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } } peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } }
I20260924 17:46:21.117616 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15414
I20260924 17:46:21.117863 15456 ts_tablet_manager.cc:1428] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 17:46:21.117887 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:39597
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:37595
--webserver_port=43503
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:21.122092 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.108700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:42701 (local address 127.1.8.190:37595)
0924 17:46:21.108960 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.108970 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.110800 (+ 1830us) server_negotiation.cc:407] Connection header received
0924 17:46:21.111637 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.111645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.111710 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.111870 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:21.112607 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.113528 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.114694 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.115204 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.115321 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.117897 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:21.117901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:21.117902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:21.117917 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:21.120090 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.120542 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.120545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.120546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.120650 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.120951 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.120952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.120952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.121266 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:21.121368 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.121731 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.121875 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260924 17:46:21.122118 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.108568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:42701)
0924 17:46:21.110554 (+ 1986us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.110630 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.111360 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.111905 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.111910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.112269 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:21.112467 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.112473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.113652 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.113654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.114585 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.114588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.115309 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.115310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.115456 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.116118 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:21.116127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:21.117768 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:21.120202 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:21.120209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:21.120211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:21.120445 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:21.120731 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:21.120732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:21.120733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:21.120876 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:21.121357 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:21.121364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:21.121538 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.121656 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.121849 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1923,"thread_start_us":98,"threads_started":1}
I20260924 17:46:21.123104 15564 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37595
I20260924 17:46:21.123207 15564 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:21.123406 15564 heartbeater.cc:511] Master 127.1.8.190:37595 requested a full tablet report, sending...
I20260924 17:46:21.124257 15376 ts_manager.cc:194] Registered new tserver with Master: 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843)
I20260924 17:46:21.125622 15376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:42701
I20260924 17:46:21.126798 15564 heartbeater.cc:503] Master 127.1.8.190:37595 was elected leader, sending a full tablet report...
I20260924 17:46:21.135617 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.124986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33843 (local address 127.0.0.1:44550)
0924 17:46:21.131386 (+ 6400us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.131427 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.131513 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.131728 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.131731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.131734 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:21.131807 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.131809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.133143 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.133144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.134439 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.134443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.135376 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.135376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.135437 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.135449 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.135508 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.135550 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6335,"thread_start_us":90,"threads_started":1}
I20260924 17:46:21.135666 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.125947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.1.8.129:33843)
0924 17:46:21.126140 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.126145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.131515 (+ 5370us) server_negotiation.cc:407] Connection header received
0924 17:46:21.131598 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.131600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.131618 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.131711 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:21.131881 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.133040 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.134522 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.135278 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.135357 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.135418 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.135525 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.135577 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260924 17:46:21.136117 15498 tablet_service.cc:1514] Processing DeleteTablet for tablet a7999e8ddede49e68fe545585aab5d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:20 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:44550
I20260924 17:46:21.136687 15573 tablet_replica.cc:330] stopping tablet replica
I20260924 17:46:21.136803 15573 raft_consensus.cc:2229] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:21.136875 15573 raft_consensus.cc:2258] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:21.137205 15573 ts_tablet_manager.cc:1905] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:21.139781 15573 ts_tablet_manager.cc:1918] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:21.139865 15573 log.cc:1196] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/a7999e8ddede49e68fe545585aab5d2d
I20260924 17:46:21.140105 15573 ts_tablet_manager.cc:1939] T a7999e8ddede49e68fe545585aab5d2d P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Deleting consensus metadata
I20260924 17:46:21.140754 15362 catalog_manager.cc:4841] TS 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843): tablet a7999e8ddede49e68fe545585aab5d2d (table test-table [id=4c2d8981ae024fada9865d2ca61ade58]) successfully deleted
W20260924 17:46:21.206913 15570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:21.207206 15570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:21.207243 15570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:21.208427 15570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:21.208514 15570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:21.208563 15570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:21.210139 15570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:39597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=43503
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:21.210464 15570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:21.210728 15570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:21.216046 15579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.216041 15581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.216041 15578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.216355 15570 server_base.cc:1034] running on GCE node
I20260924 17:46:21.216619 15570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:21.216882 15570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:21.218055 15570 hybrid_clock.cc:648] HybridClock initialized: now 1790271981218044 us; error 39 us; skew 500 ppm
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:21.220551 15570 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:21.220994 15570 webserver.cc:460] Webserver started at http://127.1.8.130:43503/ using document root <none> and password file <none>
I20260924 17:46:21.221240 15570 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:21.221326 15570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:21.222759 15570 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 17:46:21.223356 15588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.223526 15570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:46:21.223634 15570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "b901152d553af4f0ac93e01973a414d8"
server_key_iv: "c24a34e79c1ab6b8cea497c56ac641c8"
server_key_version: "encryptionkey@0"
I20260924 17:46:21.223994 15570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:21.240378 15570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:21.242357 15570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:21.242616 15570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:21.242867 15570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:21.243181 15570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:21.243214 15570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.243237 15570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:21.243283 15570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.249168 15570 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39597
I20260924 17:46:21.249253 15701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39597 every 8 connection(s)
I20260924 17:46:21.249603 15570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:46:21.254909 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15570
I20260924 17:46:21.255203 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:38035
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:37595
--webserver_port=42493
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:21.259837 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.250379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:37339 (local address 127.1.8.190:37595)
0924 17:46:21.250476 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.250478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.250686 (+ 208us) server_negotiation.cc:407] Connection header received
0924 17:46:21.251209 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.251210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.251224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.251278 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:21.252195 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.252689 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.253655 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.253850 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.253900 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.256660 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:21.256662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:21.256662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:21.256670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:21.258335 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.258772 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.258774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.258774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.258841 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.259080 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.259082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.259082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.259243 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:21.259322 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.259587 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.259694 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 17:46:21.259850 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.250368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:37339)
0924 17:46:21.250554 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.250592 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.251117 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.251336 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.251339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.251916 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:21.252113 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.252117 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.252771 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.252772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.253578 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.253581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.253925 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.253926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.254031 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.254637 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:21.254646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:21.256549 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:21.258469 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:21.258475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:21.258477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:21.258694 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:21.258910 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:21.258911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:21.258911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:21.259012 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:21.259317 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:21.259321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:21.259452 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.259540 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.259664 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260924 17:46:21.260391 15702 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37595
I20260924 17:46:21.260522 15702 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:21.260746 15702 heartbeater.cc:511] Master 127.1.8.190:37595 requested a full tablet report, sending...
I20260924 17:46:21.261194 15376 ts_manager.cc:194] Registered new tserver with Master: ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597)
I20260924 17:46:21.261581 15376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:37339
W20260924 17:46:21.339040 15705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:21.339264 15705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:21.339288 15705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:21.340466 15705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:21.340513 15705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:21.340540 15705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:21.342037 15705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:38035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=42493
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:21.342286 15705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:21.342564 15705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:21.347584 15713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.347571 15711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.347550 15710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.347599 15705 server_base.cc:1034] running on GCE node
I20260924 17:46:21.347976 15705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:21.348248 15705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:21.349395 15705 hybrid_clock.cc:648] HybridClock initialized: now 1790271981349382 us; error 29 us; skew 500 ppm
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:21.351879 15705 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:21.352275 15705 webserver.cc:460] Webserver started at http://127.1.8.131:42493/ using document root <none> and password file <none>
I20260924 17:46:21.352454 15705 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:21.352504 15705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:21.353801 15705 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:21.354359 15720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.354544 15705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:21.354651 15705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "e67af083c1f3421182e9017026a62b8f"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "c38197db996ae7cb2bad0d8b83ccae96"
server_key_iv: "071eab1c9900c154369ce6d12f7618ac"
server_key_version: "encryptionkey@0"
I20260924 17:46:21.355027 15705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:21.363969 15705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:21.365867 15705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:21.366047 15705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:21.366307 15705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:21.366647 15705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:21.366706 15705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.366752 15705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:21.366775 15705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.372763 15705 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:38035
I20260924 17:46:21.372805 15833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:38035 every 8 connection(s)
I20260924 17:46:21.373181 15705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:21.381112 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15705
I20260924 17:46:21.383144 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.374496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:38421 (local address 127.1.8.190:37595)
0924 17:46:21.374587 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.374589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.374605 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:46:21.374960 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.374961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.374970 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.375037 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:21.375947 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.376431 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.377627 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.377812 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.377871 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.380237 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:21.380239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:21.380239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:21.380246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:21.381604 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.382110 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.382111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.382111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.382167 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.382476 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.382477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.382477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.382622 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:21.382691 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.382964 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.383054 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 17:46:21.383224 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.374060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:38421)
0924 17:46:21.374257 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.374308 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.374870 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.375094 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.375097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.375674 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:21.375868 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.375873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.376535 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.376537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.377547 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.377550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.377913 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.377914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.378028 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.378624 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:21.378633 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:21.380104 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:21.381703 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:21.381710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:21.381712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:21.382036 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:21.382235 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:21.382236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:21.382237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:21.382387 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:21.382692 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:21.382697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:21.382821 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.382901 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.383027 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260924 17:46:21.383724 15834 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37595
I20260924 17:46:21.383833 15834 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:21.384058 15834 heartbeater.cc:511] Master 127.1.8.190:37595 requested a full tablet report, sending...
I20260924 17:46:21.384452 15376 ts_manager.cc:194] Registered new tserver with Master: e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
I20260924 17:46:21.384943 15376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:38421
I20260924 17:46:21.388510 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.381542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.1.8.190:37595)
0924 17:46:21.381642 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.381643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.381694 (+ 51us) server_negotiation.cc:407] Connection header received
0924 17:46:21.381770 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.381770 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.381780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.381833 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:21.382615 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.383301 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.384309 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.384604 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.384669 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.386098 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:21.386099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:21.386099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:21.386106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:21.387353 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.387715 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.387716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.387717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.387778 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:21.387988 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:21.387989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:21.387989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:21.388104 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:21.388289 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.388324 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.388403 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260924 17:46:21.389240 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:46:21.391870 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.389555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.1.8.190:37595)
0924 17:46:21.389610 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.389613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.389735 (+ 122us) server_negotiation.cc:407] Connection header received
0924 17:46:21.389781 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.389782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.389812 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.389869 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:21.389954 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.390461 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.391177 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.391363 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.391413 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.391565 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:21.391665 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:21.391727 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.391763 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.391781 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 17:46:21.392189 15376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:46:21.393874 15376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:46:21.395987 15498 tablet_service.cc:1467] Processing CreateTablet for tablet 93468583592a40c0804e9cc42603083b (DEFAULT_TABLE table=test-table [id=51759c6f300a417aa42174c468717914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:21.396217 15498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93468583592a40c0804e9cc42603083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:21.397823 15456 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap starting.
I20260924 17:46:21.398144 15456 tablet_bootstrap.cc:654] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:21.398679 15456 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: No bootstrap required, opened a new log
I20260924 17:46:21.398722 15456 ts_tablet_manager.cc:1397] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:21.398828 15456 raft_consensus.cc:348] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.398907 15456 raft_consensus.cc:374] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:21.398962 15456 raft_consensus.cc:729] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79aba118bf9f464b9c4cfd5d7e1c1ae3, State: Initialized, Role: FOLLOWER
I20260924 17:46:21.399020 15456 consensus_queue.cc:260] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.399112 15456 ts_tablet_manager.cc:1428] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.400244 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.395768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:38035 (local address 127.0.0.1:42414)
0924 17:46:21.396420 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.396430 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.396474 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.396768 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.396769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.396771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:21.396812 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.396814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.397848 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.397849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.398929 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.398931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.399967 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.399968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.400041 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.400045 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.400101 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.400141 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":482,"thread_start_us":101,"threads_started":1}
I20260924 17:46:21.400259 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.396118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.1.8.131:38035)
0924 17:46:21.396598 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.396602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.396609 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:46:21.396641 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.396642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.396654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.396890 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:21.396971 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.397730 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.399026 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.399855 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.399925 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.399991 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.400126 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.400174 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":79,"threads_started":1}
I20260924 17:46:21.400341 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.395911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.1.8.130:39597)
0924 17:46:21.396138 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.396142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.396154 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:46:21.396194 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.396196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.396213 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.396302 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:21.396439 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.397491 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.398887 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.399878 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.400014 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.400078 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.400211 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.400248 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":116,"threads_started":1}
I20260924 17:46:21.400343 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.395768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39597 (local address 127.0.0.1:47952)
0924 17:46:21.395807 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.395819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.395855 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.396317 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.396318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.396320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:21.396367 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.396369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.397606 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.397607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.398800 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.398803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.400034 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.400035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.400126 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.400130 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.400189 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.400231 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":1}
I20260924 17:46:21.401057 15768 tablet_service.cc:1467] Processing CreateTablet for tablet 93468583592a40c0804e9cc42603083b (DEFAULT_TABLE table=test-table [id=51759c6f300a417aa42174c468717914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:21.401165 15636 tablet_service.cc:1467] Processing CreateTablet for tablet 93468583592a40c0804e9cc42603083b (DEFAULT_TABLE table=test-table [id=51759c6f300a417aa42174c468717914]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:21.401417 15768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93468583592a40c0804e9cc42603083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:21.401561 15636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93468583592a40c0804e9cc42603083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:21.404237 15845 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Bootstrap starting.
I20260924 17:46:21.404302 15844 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Bootstrap starting.
I20260924 17:46:21.404868 15845 tablet_bootstrap.cc:654] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:21.404928 15844 tablet_bootstrap.cc:654] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:21.405365 15845 log.cc:824] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:21.405407 15844 log.cc:824] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:21.406112 15845 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: No bootstrap required, opened a new log
I20260924 17:46:21.406124 15844 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: No bootstrap required, opened a new log
I20260924 17:46:21.406201 15845 ts_tablet_manager.cc:1397] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:21.406213 15844 ts_tablet_manager.cc:1397] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:46:21.407840 15845 raft_consensus.cc:348] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.407899 15844 raft_consensus.cc:348] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.408015 15845 raft_consensus.cc:374] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:21.408030 15844 raft_consensus.cc:374] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:21.408062 15845 raft_consensus.cc:729] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1ad90a2a484856a5f113f4c7930a7b, State: Initialized, Role: FOLLOWER
I20260924 17:46:21.408095 15844 raft_consensus.cc:729] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67af083c1f3421182e9017026a62b8f, State: Initialized, Role: FOLLOWER
I20260924 17:46:21.408179 15845 consensus_queue.cc:260] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.408206 15844 consensus_queue.cc:260] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.408488 15702 heartbeater.cc:503] Master 127.1.8.190:37595 was elected leader, sending a full tablet report...
I20260924 17:46:21.408533 15834 heartbeater.cc:503] Master 127.1.8.190:37595 was elected leader, sending a full tablet report...
I20260924 17:46:21.408504 15845 ts_tablet_manager.cc:1428] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:21.408779 15844 ts_tablet_manager.cc:1428] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:46:21.468815 15849 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:21.468945 15849 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.469273 15849 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843)
I20260924 17:46:21.472222 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.469605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:56969 (local address 127.1.8.130:39597)
0924 17:46:21.469663 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.469665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.469778 (+ 113us) server_negotiation.cc:407] Connection header received
0924 17:46:21.469855 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.469856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.469867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.469934 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:21.470102 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.470578 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.471498 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.471870 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.471956 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.471966 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.472120 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.472155 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:46:21.472231 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.469654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:48383 (local address 127.1.8.129:33843)
0924 17:46:21.469701 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.469705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.469718 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:46:21.469743 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.469743 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.469753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.469819 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:21.469983 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.470535 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.471471 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.471838 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.471923 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.471939 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.472106 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.472144 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:46:21.472244 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.469479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33843 (local address 127.1.8.131:48383)
0924 17:46:21.469544 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.469558 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.469629 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.469824 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.469825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.469826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:21.469908 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.469909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.470625 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.470626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.471374 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.471376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.471921 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.471922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.472016 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.472019 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.472096 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.472139 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:46:21.472558 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.469491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39597 (local address 127.1.8.131:56969)
0924 17:46:21.469691 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.469701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.469790 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.469952 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:21.469953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:21.469955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:21.470028 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.470030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.470667 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.470668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.471403 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:21.471405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.471959 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.471960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:21.472038 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.472041 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:46:21.472103 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 17:46:21.472145 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":17,"thread_start_us":104,"threads_started":1}
I20260924 17:46:21.472965 15656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93468583592a40c0804e9cc42603083b" candidate_uuid: "e67af083c1f3421182e9017026a62b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" is_pre_election: true
I20260924 17:46:21.473019 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93468583592a40c0804e9cc42603083b" candidate_uuid: "e67af083c1f3421182e9017026a62b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" is_pre_election: true
I20260924 17:46:21.473156 15518 raft_consensus.cc:2454] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e67af083c1f3421182e9017026a62b8f in term 0.
I20260924 17:46:21.473156 15656 raft_consensus.cc:2454] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e67af083c1f3421182e9017026a62b8f in term 0.
I20260924 17:46:21.473379 15723 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3, e67af083c1f3421182e9017026a62b8f; no voters:
I20260924 17:46:21.473523 15849 raft_consensus.cc:2781] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:46:21.473578 15849 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:46:21.473610 15849 raft_consensus.cc:3037] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:21.474303 15849 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.474473 15849 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [CANDIDATE]: Term 1 election: Requested vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843)
I20260924 17:46:21.474658 15656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93468583592a40c0804e9cc42603083b" candidate_uuid: "e67af083c1f3421182e9017026a62b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
I20260924 17:46:21.474658 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93468583592a40c0804e9cc42603083b" candidate_uuid: "e67af083c1f3421182e9017026a62b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3"
I20260924 17:46:21.474746 15656 raft_consensus.cc:3037] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:21.474746 15518 raft_consensus.cc:3037] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:21.475678 15656 raft_consensus.cc:2454] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e67af083c1f3421182e9017026a62b8f in term 1.
I20260924 17:46:21.475678 15518 raft_consensus.cc:2454] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e67af083c1f3421182e9017026a62b8f in term 1.
I20260924 17:46:21.475847 15723 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3, e67af083c1f3421182e9017026a62b8f; no voters:
I20260924 17:46:21.475942 15849 raft_consensus.cc:2781] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:21.476119 15849 raft_consensus.cc:686] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 1 LEADER]: Becoming Leader. State: Replica: e67af083c1f3421182e9017026a62b8f, State: Running, Role: LEADER
I20260924 17:46:21.476221 15849 consensus_queue.cc:237] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.477161 15375 catalog_manager.cc:5495] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to e67af083c1f3421182e9017026a62b8f (127.1.8.131). New cstate: current_term: 1 leader_uuid: "e67af083c1f3421182e9017026a62b8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } health_report { overall_health: UNKNOWN } } }
I20260924 17:46:21.496582 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.493624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.1.8.131:38035)
0924 17:46:21.493672 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:21.493675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:21.494180 (+ 505us) server_negotiation.cc:407] Connection header received
0924 17:46:21.494272 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:21.494273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:21.494302 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:21.494382 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:21.494559 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.495025 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.495770 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:21.495964 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:21.496049 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:21.496207 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:21.496305 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:21.496394 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:21.496434 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 17:46:21.496455 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:46:21.498447 15656 raft_consensus.cc:1261] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Refusing update from remote peer e67af083c1f3421182e9017026a62b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:21.498458 15518 raft_consensus.cc:1261] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Refusing update from remote peer e67af083c1f3421182e9017026a62b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:21.498824 15849 consensus_queue.cc:1035] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:21.498968 15851 consensus_queue.cc:1035] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:46:21.500160 15703 tablet.cc:2369] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:46:21.501638 15854 mvcc.cc:205] Tried to move back new op lower bound from 7332954036214738944 to 7332954036127293440. Current Snapshot: MvccSnapshot[applied={T|T < 7332954036214738944}]
I20260924 17:46:21.502558 15856 mvcc.cc:205] Tried to move back new op lower bound from 7332954036214738944 to 7332954036127293440. Current Snapshot: MvccSnapshot[applied={T|T < 7332954036214738944 or (T in {7332954036214738944})}]
I20260924 17:46:21.504033 15376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:21.504128 15376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:21.504887 15376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Sending DeleteTablet for 3 replicas of tablet 93468583592a40c0804e9cc42603083b
I20260924 17:46:21.505228 15768 tablet_service.cc:1514] Processing DeleteTablet for tablet 93468583592a40c0804e9cc42603083b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:21 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42414
I20260924 17:46:21.505260 15636 tablet_service.cc:1514] Processing DeleteTablet for tablet 93468583592a40c0804e9cc42603083b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:21 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:47952
I20260924 17:46:21.505308 15498 tablet_service.cc:1514] Processing DeleteTablet for tablet 93468583592a40c0804e9cc42603083b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:21 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:44550
I20260924 17:46:21.505402 15573 tablet_replica.cc:330] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: stopping tablet replica
I20260924 17:46:21.505445 15573 raft_consensus.cc:2229] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:21.505486 15573 raft_consensus.cc:2258] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:21.505476 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15414
I20260924 17:46:21.507401 15864 tablet_replica.cc:330] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: stopping tablet replica
I20260924 17:46:21.507520 15864 raft_consensus.cc:2229] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:46:21.507654 15864 raft_consensus.cc:2258] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:21.507830 15865 tablet_replica.cc:330] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: stopping tablet replica
I20260924 17:46:21.507938 15865 raft_consensus.cc:2229] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:21.508062 15865 raft_consensus.cc:2258] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:21.508358 15864 ts_tablet_manager.cc:1905] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:21.508510 15865 ts_tablet_manager.cc:1905] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:21.510335 15865 ts_tablet_manager.cc:1918] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:21.510452 15865 log.cc:1196] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/93468583592a40c0804e9cc42603083b
I20260924 17:46:21.510677 15864 ts_tablet_manager.cc:1918] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:21.510757 15865 ts_tablet_manager.cc:1939] T 93468583592a40c0804e9cc42603083b P ed1ad90a2a484856a5f113f4c7930a7b: Deleting consensus metadata
I20260924 17:46:21.510846 15864 log.cc:1196] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/93468583592a40c0804e9cc42603083b
I20260924 17:46:21.511164 15864 ts_tablet_manager.cc:1939] T 93468583592a40c0804e9cc42603083b P e67af083c1f3421182e9017026a62b8f: Deleting consensus metadata
I20260924 17:46:21.511705 15363 catalog_manager.cc:4841] TS e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): tablet 93468583592a40c0804e9cc42603083b (table test-table [id=51759c6f300a417aa42174c468717914]) successfully deleted
W20260924 17:46:21.512210 15362 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:33843 (error 108)
I20260924 17:46:21.512595 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15570
I20260924 17:46:21.512663 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.512549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33843 (local address 127.0.0.1:44556)
0924 17:46:21.512591 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.512627 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 17:46:21.513526 15362 catalog_manager.cc:4584] TS 79aba118bf9f464b9c4cfd5d7e1c1ae3 (127.1.8.129:33843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93468583592a40c0804e9cc42603083b: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33843: connect: Connection refused (error 111)
I20260924 17:46:21.518460 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.518309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39597 (local address 127.0.0.1:47954)
0924 17:46:21.518370 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.518420 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.130:39597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 17:46:21.518532 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15705
I20260924 17:46:21.523788 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15343
I20260924 17:46:21.529443 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37595
--webserver_interface=127.1.8.190
--webserver_port=45243
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:21.611781 15867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:21.612016 15867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:21.612040 15867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:21.613310 15867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:21.613359 15867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:21.613374 15867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:21.613415 15867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:21.613442 15867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:21.615080 15867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:37595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=45243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:21.615460 15867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:21.615754 15867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:21.621158 15872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.621186 15875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.621346 15867 server_base.cc:1034] running on GCE node
W20260924 17:46:21.621158 15873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.621728 15867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:21.622058 15867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:21.623230 15867 hybrid_clock.cc:648] HybridClock initialized: now 1790271981623218 us; error 46 us; skew 500 ppm
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:21.625730 15867 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:21.626137 15867 webserver.cc:460] Webserver started at http://127.1.8.190:45243/ using document root <none> and password file <none>
I20260924 17:46:21.626327 15867 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:21.626367 15867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:21.627730 15867 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:21.628358 15882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.628561 15867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:21.628646 15867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "68843f529c1f4f15addf79eb7f10b9cd"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "020d5050e60b2d59c3de101b9076a6f3"
server_key_iv: "2bcd045ff84b8065a5978c3562a288cb"
server_key_version: "encryptionkey@0"
I20260924 17:46:21.628984 15867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:21.643899 15867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:21.646023 15867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:21.646145 15867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:21.650305 15867 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:37595
I20260924 17:46:21.650358 15934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:37595 every 8 connection(s)
I20260924 17:46:21.650895 15867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:21.652547 15935 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:46:21.653281 15935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap starting.
I20260924 17:46:21.654739 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15867
I20260924 17:46:21.655007 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:33843
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:37595
--webserver_port=33425
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:21.662330 15935 log.cc:824] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:21.664765 15935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:21.665123 15935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd: Bootstrap complete.
I20260924 17:46:21.667493 15935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:21.667723 15935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68843f529c1f4f15addf79eb7f10b9cd, State: Initialized, Role: FOLLOWER
I20260924 17:46:21.667909 15935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:21.667973 15935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:21.668002 15935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:21.668073 15935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 2 FOLLOWER]: Advancing to term 3
I20260924 17:46:21.669106 15935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:21.669241 15935 leader_election.cc:302] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68843f529c1f4f15addf79eb7f10b9cd; no voters:
I20260924 17:46:21.669494 15935 leader_election.cc:288] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 17:46:21.669529 15939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 17:46:21.669647 15939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [term 3 LEADER]: Becoming Leader. State: Replica: 68843f529c1f4f15addf79eb7f10b9cd, State: Running, Role: LEADER
I20260924 17:46:21.669745 15939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } }
I20260924 17:46:21.669780 15935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:21.670022 15939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68843f529c1f4f15addf79eb7f10b9cd. Latest consensus state: current_term: 3 leader_uuid: "68843f529c1f4f15addf79eb7f10b9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } } }
I20260924 17:46:21.670089 15939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:21.670161 15941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "68843f529c1f4f15addf79eb7f10b9cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68843f529c1f4f15addf79eb7f10b9cd" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37595 } } }
I20260924 17:46:21.670209 15941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68843f529c1f4f15addf79eb7f10b9cd [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:21.670362 15943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:21.671183 15943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:21.671288 15943 catalog_manager.cc:1259] Loaded cluster ID: f441ac147c514b2f8e77554bca7c5ab7
I20260924 17:46:21.671326 15943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 17:46:21.671633 15943 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7010a5a9ad google::LogMessage::Fail() at ??:0
@ 0x7f7010a5d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7010a5a4de google::LogMessage::Flush() at ??:0
@ 0x7f7010a5ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f701897b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f70189848b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f70129280d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f701291d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70137816db start_thread at ??:0
@ 0x7f700f55861f clone at ??:0
W20260924 17:46:21.752015 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:21.752221 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:21.752246 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:21.753443 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:21.753486 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:21.753522 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:21.755048 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:33843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=33425
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:21.755297 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:21.755548 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:21.760668 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.760726 15974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.760936 15972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.760947 15937 server_base.cc:1034] running on GCE node
I20260924 17:46:21.761356 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:21.761636 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:21.762826 15937 hybrid_clock.cc:648] HybridClock initialized: now 1790271981762805 us; error 40 us; skew 500 ppm
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:21.765739 15937 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:21.766171 15937 webserver.cc:460] Webserver started at http://127.1.8.129:33425/ using document root <none> and password file <none>
I20260924 17:46:21.766460 15937 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:21.766532 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:21.768028 15937 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:21.768788 15981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.769025 15937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:21.769129 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "ac8a5f8b95db50d93752690e5172e569"
server_key_iv: "e1d27cb7c036974f4acdcf1450e1ad9c"
server_key_version: "encryptionkey@0"
I20260924 17:46:21.769481 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:21.779810 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:21.782567 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:21.782733 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:21.782972 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:21.783454 15988 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:46:21.785377 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:46:21.785439 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:46:21.785501 15937 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:46:21.786137 15937 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:46:21.786196 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:21.786317 15988 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap starting.
I20260924 17:46:21.793475 15937 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:33843
I20260924 17:46:21.793565 16095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:33843 every 8 connection(s)
I20260924 17:46:21.793948 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:21.797593 15988 log.cc:824] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:21.800036 15988 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:21.800365 15988 tablet_bootstrap.cc:492] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Bootstrap complete.
I20260924 17:46:21.800776 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 15937
I20260924 17:46:21.801049 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:39597
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:37595
--webserver_port=43503
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:21.801491 15988 ts_tablet_manager.cc:1397] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260924 17:46:21.803160 15988 raft_consensus.cc:348] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.803536 15988 raft_consensus.cc:729] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79aba118bf9f464b9c4cfd5d7e1c1ae3, State: Initialized, Role: FOLLOWER
I20260924 17:46:21.803690 15988 consensus_queue.cc:260] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:21.804114 15988 ts_tablet_manager.cc:1428] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260924 17:46:21.908587 16100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:21.908838 16100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:21.908866 16100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:21.910130 16100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:21.910181 16100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:21.910209 16100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:21.911980 16100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:39597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=43503
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:21.912247 16100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:21.912514 16100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:21.918234 16106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:21.918222 16109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.918479 16100 server_base.cc:1034] running on GCE node
W20260924 17:46:21.918215 16107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:21.918769 16100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:21.919059 16100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:21.920218 16100 hybrid_clock.cc:648] HybridClock initialized: now 1790271981920042 us; error 188 us; skew 500 ppm
Sep 24 17:46:21 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:21.922931 16100 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:21.923429 16100 webserver.cc:460] Webserver started at http://127.1.8.130:43503/ using document root <none> and password file <none>
W20260924 17:46:21.923458 16098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:46:21.796268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:48581)
0924 17:46:21.796476 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.796543 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:21.797304 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:21.923277 (+125973us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
W20260924 17:46:21.923735 16096 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:37595 (0 consecutive failures): Network error: Failed to ping master at 127.1.8.190:37595: Client connection negotiation failed: client connection to 127.1.8.190:37595: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:46:21.923728 16100 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:21.923799 16100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:21.924032 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.923930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:58043)
0924 17:46:21.923975 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.924013 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 17:46:21.924360 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.924259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:35355)
0924 17:46:21.924313 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.924339 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 17:46:21.924489 16096 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:46:21.925323 16100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:21.925884 16116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.926038 16100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.926121 16100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "ed1ad90a2a484856a5f113f4c7930a7b"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "b901152d553af4f0ac93e01973a414d8"
server_key_iv: "c24a34e79c1ab6b8cea497c56ac641c8"
server_key_version: "encryptionkey@0"
I20260924 17:46:21.926534 16100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:21.944116 16100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:21.946599 16100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:21.946775 16100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:21.947034 16100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:21.947343 16100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:21.947377 16100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.947422 16100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:21.947448 16100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:21.953174 16100 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39597
I20260924 17:46:21.953254 16229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39597 every 8 connection(s)
I20260924 17:46:21.953631 16100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:46:21.954794 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.954341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:41437)
0924 17:46:21.954567 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.954645 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":131,"threads_started":1}
W20260924 17:46:21.955227 16230 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:37595 (0 consecutive failures): Network error: Failed to ping master at 127.1.8.190:37595: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
I20260924 17:46:21.955521 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.955426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:47415)
0924 17:46:21.955472 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.955500 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 17:46:21.955847 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:21.955765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:53609)
0924 17:46:21.955809 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:21.955831 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:46:21.956066 16230 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:46:21.956954 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16100
I20260924 17:46:21.957206 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:38035
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:37595
--webserver_port=42493
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:22.040481 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:22.040763 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:22.040813 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:22.042023 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:22.042100 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:22.042146 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:22.043684 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:38035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=42493
--tserver_master_addrs=127.1.8.190:37595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:22.043977 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:22.044231 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:22.049368 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:22.049371 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:22.049444 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:22.049669 16233 server_base.cc:1034] running on GCE node
I20260924 17:46:22.049928 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:22.050239 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:22.051419 16233 hybrid_clock.cc:648] HybridClock initialized: now 1790271982051410 us; error 40 us; skew 500 ppm
Sep 24 17:46:22 dist-test-slave-r7hf krb5kdc[14797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271982, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:22.054086 16233 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:22.054574 16233 webserver.cc:460] Webserver started at http://127.1.8.131:42493/ using document root <none> and password file <none>
I20260924 17:46:22.054764 16233 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:22.054806 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:22.056147 16233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:22.056738 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:22.056895 16233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:46:22.056985 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "e67af083c1f3421182e9017026a62b8f"
format_stamp: "Formatted at 2026-09-24 17:46:20 on dist-test-slave-r7hf"
server_key: "c38197db996ae7cb2bad0d8b83ccae96"
server_key_iv: "071eab1c9900c154369ce6d12f7618ac"
server_key_version: "encryptionkey@0"
I20260924 17:46:22.057330 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:22.074967 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:22.076954 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:22.077073 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:22.077258 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:22.077594 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:22.077629 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:22.077677 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:22.077702 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:22.083460 16233 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:38035
I20260924 17:46:22.083542 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:38035 every 8 connection(s)
I20260924 17:46:22.083894 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:46:22.084981 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:22.084590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:50661)
0924 17:46:22.084764 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:22.084839 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
W20260924 17:46:22.085420 16362 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:37595 (0 consecutive failures): Network error: Failed to ping master at 127.1.8.190:37595: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
I20260924 17:46:22.085717 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:22.085621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:60035)
0924 17:46:22.085671 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:22.085698 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:46:22.086009 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:22.085930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:46139)
0924 17:46:22.085969 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:22.085995 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 17:46:22.086126 16362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:46:22.092356 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16233
I20260924 17:46:22.925232 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:22.924949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:55345)
0924 17:46:22.925080 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:22.925128 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 17:46:22.956638 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:22.956402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:53353)
0924 17:46:22.956541 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:22.956585 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260924 17:46:23.086753 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.086534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:36769)
0924 17:46:23.086653 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:23.086697 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260924 17:46:23.530301 16369 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:23.530512 16369 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:23.530922 16369 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
Sep 24 17:46:23 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.130@KRBTEST.COM
Sep 24 17:46:23 dist-test-slave-r7hf krb5kdc[14797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271981, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.131@KRBTEST.COM
I20260924 17:46:23.544176 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.531126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39597 (local address 127.1.8.129:57523)
0924 17:46:23.531327 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:23.531337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:23.531384 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:23.532332 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:23.532335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:23.532848 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:23.533146 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:23.533155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.534848 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.534849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:23.536234 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:23.536237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.537109 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.537110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:23.537246 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:23.537945 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:23.537957 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:23.539940 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:23.542221 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:23.542230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:23.542233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:23.542640 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:23.542929 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:23.542930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:23.542930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:23.543086 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:23.543572 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:23.543578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:23.543728 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 17:46:23.543839 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 17:46:23.544061 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":109,"thread_start_us":104,"threads_started":1}
I20260924 17:46:23.544268 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.531236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:50645 (local address 127.1.8.131:38035)
0924 17:46:23.531406 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:23.531419 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:23.531452 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:46:23.532025 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:23.532028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:23.532078 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:23.532246 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:23.533285 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.534729 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.536336 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.537014 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.537116 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:23.540387 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:23.540391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:23.540391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:23.540402 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:23.542622 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:23.543040 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:23.543043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:23.543043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:23.543166 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:23.543365 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:23.543366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:23.543366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:23.543636 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:23.543718 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:23.544016 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 17:46:23.544145 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260924 17:46:23.544277 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.531273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:57523 (local address 127.1.8.130:39597)
0924 17:46:23.531409 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:23.531419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:23.531445 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:46:23.532017 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:23.532020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:23.532078 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:23.532245 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:23.533284 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.534729 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.536337 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.536991 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.537116 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:23.540107 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:23.540113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:23.540113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:23.540128 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:23.542138 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:23.542729 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:23.542733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:23.542733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:23.542851 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:23.543162 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:23.543163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:23.543163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:23.543504 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:23.543565 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:23.544006 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 17:46:23.544159 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":49,"threads_started":1}
I20260924 17:46:23.544294 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.531115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:38035 (local address 127.1.8.129:50645)
0924 17:46:23.531282 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:23.531295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:23.531371 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:23.532332 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:23.532335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:23.532849 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:23.533146 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:23.533155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.534857 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.534857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:23.536234 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:23.536237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:23.537109 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:23.537110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:23.537246 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:23.537952 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:23.537959 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:23.540264 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:23.542719 (+ 2455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:23.542720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:23.542721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:23.542955 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:23.543237 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:23.543238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:23.543238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:23.543298 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:23.543730 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:23.543731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:23.543773 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 17:46:23.543942 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 17:46:23.544160 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":150,"thread_start_us":54,"threads_started":1}
W20260924 17:46:23.545455 15982 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
W20260924 17:46:23.545600 15985 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
I20260924 17:46:23.545704 15985 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3; no voters: e67af083c1f3421182e9017026a62b8f, ed1ad90a2a484856a5f113f4c7930a7b
I20260924 17:46:23.545907 16369 raft_consensus.cc:2726] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:46:23.926036 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.925866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:44047)
0924 17:46:23.925952 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:23.925995 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 17:46:23.957357 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:23.957089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:52051)
0924 17:46:23.957253 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:23.957307 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260924 17:46:24.087384 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:24.087179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:42207)
0924 17:46:24.087299 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:24.087340 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 17:46:24.927007 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:24.926776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:54073)
0924 17:46:24.926911 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:24.926960 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260924 17:46:24.958153 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:24.957875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:48467)
0924 17:46:24.958069 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:24.958108 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":126,"threads_started":1}
I20260924 17:46:25.088158 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:25.087910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:49543)
0924 17:46:25.088065 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:25.088113 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260924 17:46:25.338456 16379 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:25.338588 16379 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:25.338771 16379 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
W20260924 17:46:25.339272 15985 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
W20260924 17:46:25.339355 15982 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
I20260924 17:46:25.339452 15982 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3; no voters: e67af083c1f3421182e9017026a62b8f, ed1ad90a2a484856a5f113f4c7930a7b
I20260924 17:46:25.339644 16379 raft_consensus.cc:2726] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:46:25.927738 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:25.927521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:36911)
0924 17:46:25.927649 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:25.927690 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260924 17:46:25.958864 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:25.958626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:39811)
0924 17:46:25.958773 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:25.958819 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260924 17:46:26.088933 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:26.088668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:33679)
0924 17:46:26.088836 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:26.088885 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260924 17:46:26.928670 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:26.928433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:49989)
0924 17:46:26.928582 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:26.928622 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260924 17:46:26.959619 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:26.959349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:40601)
0924 17:46:26.959530 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:26.959575 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
I20260924 17:46:27.089799 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:27.089564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:34201)
0924 17:46:27.089703 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:27.089752 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260924 17:46:27.637208 16386 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:27.637341 16386 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:27.637548 16386 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
W20260924 17:46:27.637909 15982 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
W20260924 17:46:27.638041 15985 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
I20260924 17:46:27.638116 15985 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3; no voters: e67af083c1f3421182e9017026a62b8f, ed1ad90a2a484856a5f113f4c7930a7b
I20260924 17:46:27.638223 16386 raft_consensus.cc:2726] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:46:27.929512 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:27.929251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:55323)
0924 17:46:27.929414 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:27.929462 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260924 17:46:27.960408 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:27.960145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:45961)
0924 17:46:27.960277 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:27.960354 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 17:46:28.090699 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:28.090443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:50203)
0924 17:46:28.090601 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:28.090651 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260924 17:46:28.930320 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:28.930082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:43867)
0924 17:46:28.930235 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:28.930271 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260924 17:46:28.961154 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:28.960909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:46409)
0924 17:46:28.961070 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:28.961107 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260924 17:46:29.091545 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:29.091302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:53527)
0924 17:46:29.091443 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:29.091486 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260924 17:46:29.278867 16393 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:29.279003 16393 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:29.279192 16393 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
W20260924 17:46:29.279656 15985 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
W20260924 17:46:29.279742 15982 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
I20260924 17:46:29.279776 15982 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3; no voters: e67af083c1f3421182e9017026a62b8f, ed1ad90a2a484856a5f113f4c7930a7b
I20260924 17:46:29.279870 16393 raft_consensus.cc:2726] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:46:29.931049 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:29.930822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:45769)
0924 17:46:29.930957 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:29.931001 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260924 17:46:29.961972 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:29.961666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:56179)
0924 17:46:29.961846 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:29.961921 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260924 17:46:30.092365 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:30.092082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:37451)
0924 17:46:30.092269 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:30.092317 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260924 17:46:30.931779 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:30.931552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:36543)
0924 17:46:30.931684 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:30.931731 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260924 17:46:30.962801 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:30.962568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:39407)
0924 17:46:30.962703 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:30.962751 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 17:46:31.093026 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:31.092807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:44429)
0924 17:46:31.092934 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:31.092976 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 17:46:31.932552 16400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:31.932316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:37057)
0924 17:46:31.932464 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:31.932503 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260924 17:46:31.963716 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:31.963489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:37911)
0924 17:46:31.963632 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:31.963666 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260924 17:46:32.093833 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:32.093587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:50625)
0924 17:46:32.093739 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:32.093784 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260924 17:46:32.702175 16403 raft_consensus.cc:482] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:32.702286 16403 raft_consensus.cc:504] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1ad90a2a484856a5f113f4c7930a7b" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39597 } } peers { permanent_uuid: "e67af083c1f3421182e9017026a62b8f" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 38035 } } peers { permanent_uuid: "79aba118bf9f464b9c4cfd5d7e1c1ae3" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33843 } }
I20260924 17:46:32.702631 16403 leader_election.cc:288] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597), e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035)
W20260924 17:46:32.703029 15985 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e67af083c1f3421182e9017026a62b8f (127.1.8.131:38035): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
W20260924 17:46:32.703120 15982 leader_election.cc:341] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed1ad90a2a484856a5f113f4c7930a7b (127.1.8.130:39597): Not found: Tablet not found: 93468583592a40c0804e9cc42603083b
I20260924 17:46:32.703191 15982 leader_election.cc:302] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79aba118bf9f464b9c4cfd5d7e1c1ae3; no voters: e67af083c1f3421182e9017026a62b8f, ed1ad90a2a484856a5f113f4c7930a7b
I20260924 17:46:32.703289 16403 raft_consensus.cc:2726] T 93468583592a40c0804e9cc42603083b P 79aba118bf9f464b9c4cfd5d7e1c1ae3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:46:32.933383 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:32.933138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:42061)
0924 17:46:32.933289 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:32.933334 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260924 17:46:32.964545 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:32.964304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:32887)
0924 17:46:32.964463 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:32.964496 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260924 17:46:33.094794 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:33.094499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:53997)
0924 17:46:33.094682 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:33.094736 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260924 17:46:33.934206 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:33.933975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:48551)
0924 17:46:33.934111 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:33.934159 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 17:46:33.965377 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:33.965093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:52755)
0924 17:46:33.965275 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:33.965326 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260924 17:46:34.095618 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:34.095367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:56439)
0924 17:46:34.095520 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:34.095568 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260924 17:46:34.934974 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:34.934707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:54137)
0924 17:46:34.934877 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:34.934927 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260924 17:46:34.966279 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:34.966035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:39937)
0924 17:46:34.966184 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:34.966231 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260924 17:46:35.096508 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:35.096213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:33675)
0924 17:46:35.096403 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:35.096454 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260924 17:46:35.935848 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:35.935566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:33267)
0924 17:46:35.935728 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:35.935788 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260924 17:46:35.967136 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:35.966864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:52277)
0924 17:46:35.967039 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:35.967088 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260924 17:46:36.097422 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:36.097136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:39569)
0924 17:46:36.097317 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:36.097371 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260924 17:46:36.936900 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:36.936607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.129:33339)
0924 17:46:36.936792 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:36.936844 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260924 17:46:36.968144 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:36.967845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.130:44921)
0924 17:46:36.968056 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:36.968096 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":142,"threads_started":1}
I20260924 17:46:37.098359 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:37.098110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37595 (local address 127.1.8.131:40857)
0924 17:46:37.098260 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:37.098307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260924 17:46:37.100136 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 15937
I20260924 17:46:37.105012 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16100
I20260924 17:46:37.109620 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16233
2026-09-24T17:46:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18268 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:46:37 dist-test-slave-r7hf krb5kdc[16421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:46:37 dist-test-slave-r7hf krb5kdc[16421](info): set up 2 sockets
Sep 24 17:46:37 dist-test-slave-r7hf krb5kdc[16421](info): commencing operation
krb5kdc: starting...
W20260924 17:46:38.170728 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:46:38.302178 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40109
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:38.385821 16434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:38.386056 16434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:38.386080 16434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:38.387336 16434 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 17:46:38.387382 16434 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 17:46:38.387394 16434 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 17:46:38.387406 16434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:38.387419 16434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:38.387472 16434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:38.387504 16434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:38.387521 16434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:38.389076 16434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:40109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:38.389375 16434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:38.389621 16434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:38.395252 16439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.395382 16440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:38.395478 16434 server_base.cc:1034] running on GCE node
W20260924 17:46:38.395284 16442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:38.395835 16434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:38.396157 16434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:38.397310 16434 hybrid_clock.cc:648] HybridClock initialized: now 1790271998397288 us; error 49 us; skew 500 ppm
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:38.399976 16434 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:38.400488 16434 webserver.cc:460] Webserver started at http://127.1.8.190:34889/ using document root <none> and password file <none>
I20260924 17:46:38.400695 16434 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:38.400738 16434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:38.400833 16434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:38.401894 16434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "ffc84deea31c475ba7d5cccaba11ab04"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "324e25868a082628e46b5b39abc1acad"
server_key_iv: "43afa7c60bd46555372fdd182404e95e"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.402210 16434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "ffc84deea31c475ba7d5cccaba11ab04"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "324e25868a082628e46b5b39abc1acad"
server_key_iv: "43afa7c60bd46555372fdd182404e95e"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.403731 16434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:46:38.404593 16449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.404806 16434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:38.404889 16434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "ffc84deea31c475ba7d5cccaba11ab04"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "324e25868a082628e46b5b39abc1acad"
server_key_iv: "43afa7c60bd46555372fdd182404e95e"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.404953 16434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:38.421543 16434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:38.423601 16434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:38.423732 16434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:38.427982 16434 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:40109
I20260924 17:46:38.428017 16501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:40109 every 8 connection(s)
I20260924 17:46:38.428408 16434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:38.429353 16502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:38.431883 16502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap starting.
I20260924 17:46:38.432456 16502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:38.432705 16502 log.cc:824] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:38.433435 16502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: No bootstrap required, opened a new log
I20260924 17:46:38.435068 16502 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:38.435218 16502 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:38.435241 16502 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc84deea31c475ba7d5cccaba11ab04, State: Initialized, Role: FOLLOWER
I20260924 17:46:38.435350 16502 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:38.435449 16502 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:38.435487 16502 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:38.435545 16502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:38.436120 16502 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:38.436254 16502 leader_election.cc:302] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc84deea31c475ba7d5cccaba11ab04; no voters:
I20260924 17:46:38.436453 16502 leader_election.cc:288] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:46:38.436504 16505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:38.436728 16505 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 LEADER]: Becoming Leader. State: Replica: ffc84deea31c475ba7d5cccaba11ab04, State: Running, Role: LEADER
I20260924 17:46:38.436748 16502 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:38.436823 16505 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:38.437343 16507 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc84deea31c475ba7d5cccaba11ab04. Latest consensus state: current_term: 1 leader_uuid: "ffc84deea31c475ba7d5cccaba11ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } } }
I20260924 17:46:38.437467 16507 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:38.437779 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16434
I20260924 17:46:38.437927 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:46:38.438161 1058 external_mini_cluster.cc:1428] Setting key 18640faca0220c02ce41711381eb8687
I20260924 17:46:38.437318 16506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc84deea31c475ba7d5cccaba11ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } } }
I20260924 17:46:38.438730 16506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: This master's current role is: LEADER
W20260924 17:46:38.438751 16521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:46:38.438804 16521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:46:38.438884 16514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:38.439040 16514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:38.440753 16514 catalog_manager.cc:1347] Generated new cluster ID: d6a41c748cbe44b4914ae35a76bd662c
I20260924 17:46:38.440810 16514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:38.449162 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:38.438695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.1.8.190:40109)
0924 17:46:38.439233 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:38.439241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:38.439267 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:46:38.439690 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:38.439699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:38.439767 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:38.439947 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:38.440723 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.441706 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.442617 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.443071 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.443221 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:38.445656 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:38.445661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:38.445661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:38.445675 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:38.447544 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:38.447953 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:38.447957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:38.447957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:38.448056 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:38.448266 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:38.448267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:38.448268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:38.448536 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:38.448694 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:38.448809 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 17:46:38.448968 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":76,"threads_started":1}
I20260924 17:46:38.480634 16514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:46:38.481218 16514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:38.484694 16514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Generated new TSK 0
I20260924 17:46:38.484920 16514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:46:38.529459 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40109
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:38.646862 16528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:38.647099 16528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:38.647127 16528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:38.648440 16528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:38.648499 16528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:38.648530 16528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:38.650154 16528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:38.650498 16528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:38.650764 16528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:46:38.656281 16528 server_base.cc:1034] running on GCE node
W20260924 17:46:38.656250 16533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.656246 16536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.656232 16534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:38.656662 16528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:38.656935 16528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:38.658115 16528 hybrid_clock.cc:648] HybridClock initialized: now 1790271998658093 us; error 38 us; skew 500 ppm
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:38.660925 16528 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:38.661494 16528 webserver.cc:460] Webserver started at http://127.1.8.129:39785/ using document root <none> and password file <none>
I20260924 17:46:38.661715 16528 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:38.661760 16528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:38.661859 16528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:38.663079 16528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "bc232f210a132d0767346f9609a62191"
server_key_iv: "d0a7f5fc986127d52d1bf0cb130a1541"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.663427 16528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "bc232f210a132d0767346f9609a62191"
server_key_iv: "d0a7f5fc986127d52d1bf0cb130a1541"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.664865 16528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:46:38.665642 16543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.665853 16528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:38.665946 16528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "bc232f210a132d0767346f9609a62191"
server_key_iv: "d0a7f5fc986127d52d1bf0cb130a1541"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.666009 16528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:38.687848 16528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:38.690472 16528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:38.690680 16528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:38.691021 16528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:38.691485 16528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:38.691550 16528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.691586 16528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:38.691622 16528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.699107 16528 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42907
I20260924 17:46:38.699184 16656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42907 every 8 connection(s)
I20260924 17:46:38.699548 16528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:38.706101 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16528
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:38.706243 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:46:38.706532 1058 external_mini_cluster.cc:1428] Setting key 9609050b2039072d4d1e45bc238c0bbb
I20260924 17:46:38.710279 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:38.700425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49799 (local address 127.1.8.190:40109)
0924 17:46:38.700496 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:38.700498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:38.700691 (+ 193us) server_negotiation.cc:407] Connection header received
0924 17:46:38.701188 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:38.701189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:38.701203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:38.701268 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:38.702331 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.703008 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.704077 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.704283 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.704339 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:38.706864 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:38.706866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:38.706866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:38.706874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:38.708684 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:38.709143 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:38.709145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:38.709146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:38.709220 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:38.709475 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:38.709477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:38.709477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:38.709642 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:38.709721 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:38.710028 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 17:46:38.710132 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 17:46:38.710294 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:38.700360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:49799)
0924 17:46:38.700549 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:38.700594 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:38.701098 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:38.701380 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:38.701383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:38.702026 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:38.702232 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:38.702237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.703114 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.703115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:38.703991 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:38.703993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.704366 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.704367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:38.704475 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:38.705110 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:38.705119 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:38.706733 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:38.708783 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:38.708791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:38.708793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:38.709057 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:38.709295 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:38.709296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:38.709296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:38.709407 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:38.709714 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:38.709718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:38.709882 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 17:46:38.709973 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 17:46:38.710113 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260924 17:46:38.711144 16657 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40109
I20260924 17:46:38.711278 16657 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:38.711512 16657 heartbeater.cc:511] Master 127.1.8.190:40109 requested a full tablet report, sending...
I20260924 17:46:38.712116 16466 ts_manager.cc:194] Registered new tserver with Master: c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907)
I20260924 17:46:38.712770 16466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:49799
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:46:38.734608 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40109
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:38.817683 16662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:38.817914 16662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:38.817938 16662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:38.819185 16662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:38.819236 16662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:38.819264 16662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:38.820775 16662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:38.821056 16662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:38.821303 16662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:46:38.826879 16662 server_base.cc:1034] running on GCE node
W20260924 17:46:38.826828 16667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.826825 16668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.826853 16670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:38.827282 16662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:38.827565 16662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:38.828739 16662 hybrid_clock.cc:648] HybridClock initialized: now 1790271998828722 us; error 54 us; skew 500 ppm
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:38.831604 16662 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:38.832083 16662 webserver.cc:460] Webserver started at http://127.1.8.130:35233/ using document root <none> and password file <none>
I20260924 17:46:38.832329 16662 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:38.832404 16662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:38.832556 16662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:38.833693 16662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "99b7362a03734b45a9d4fdf0871ca7e0"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "d815f79803aa3e9b964c697052c4fabc"
server_key_iv: "d291251926619ba57a015aef566509a6"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.834084 16662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "99b7362a03734b45a9d4fdf0871ca7e0"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "d815f79803aa3e9b964c697052c4fabc"
server_key_iv: "d291251926619ba57a015aef566509a6"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.835580 16662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:46:38.836501 16677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.836791 16662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:38.836930 16662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "99b7362a03734b45a9d4fdf0871ca7e0"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "d815f79803aa3e9b964c697052c4fabc"
server_key_iv: "d291251926619ba57a015aef566509a6"
server_key_version: "encryptionkey@0"
I20260924 17:46:38.837028 16662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:38.856448 16662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:38.858547 16662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:38.858719 16662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:38.858979 16662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:38.859298 16662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:38.859364 16662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.859411 16662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:38.859433 16662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:38.865533 16662 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:46635
I20260924 17:46:38.865585 16790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:46635 every 8 connection(s)
I20260924 17:46:38.865957 16662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:46:38.870229 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16662
I20260924 17:46:38.870316 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:46:38.870558 1058 external_mini_cluster.cc:1428] Setting key f23fddb2298014b1bc66435a78eed096
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:38.876909 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:38.866787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:32821 (local address 127.1.8.190:40109)
0924 17:46:38.866882 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:38.866885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:38.867087 (+ 202us) server_negotiation.cc:407] Connection header received
0924 17:46:38.867649 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:38.867650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:38.867664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:38.867716 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:38.868824 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.869418 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.870760 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.870974 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.871053 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:38.873515 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:38.873517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:38.873517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:38.873529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:38.875341 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:38.875804 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:38.875806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:38.875806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:38.875887 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:38.876169 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:38.876171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:38.876171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:38.876305 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:38.876390 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:38.876708 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 17:46:38.876804 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:46:38.877106 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:38.866750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:32821)
0924 17:46:38.866943 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:38.866992 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:38.867525 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:38.867791 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:38.867795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:38.868441 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:38.868714 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:38.868720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.869528 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.869529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:38.870659 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:38.870662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:38.871058 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:38.871060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:38.871194 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:38.871778 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:38.871786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:38.873403 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:38.875449 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:38.875458 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:38.875460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:38.875715 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:38.875962 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:38.875963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:38.875963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:38.876078 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:38.876399 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:38.876407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:38.876555 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 17:46:38.876669 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 17:46:38.876852 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260924 17:46:38.877655 16791 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40109
I20260924 17:46:38.877780 16791 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:38.877995 16791 heartbeater.cc:511] Master 127.1.8.190:40109 requested a full tablet report, sending...
I20260924 17:46:38.878443 16466 ts_manager.cc:194] Registered new tserver with Master: 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635)
I20260924 17:46:38.878850 16466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:32821
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:46:38.901163 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40109
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:38.984313 16796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:38.984556 16796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:38.984583 16796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:38.985846 16796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:38.985903 16796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:38.985932 16796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:38.987493 16796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:38.987709 16796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:38.987905 16796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:38.993284 16802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.993278 16804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:38.993284 16801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:38.993525 16796 server_base.cc:1034] running on GCE node
I20260924 17:46:38.993772 16796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:38.994040 16796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:38.995213 16796 hybrid_clock.cc:648] HybridClock initialized: now 1790271998995197 us; error 30 us; skew 500 ppm
Sep 24 17:46:38 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:38.997788 16796 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:38.998320 16796 webserver.cc:460] Webserver started at http://127.1.8.131:39303/ using document root <none> and password file <none>
I20260924 17:46:38.998593 16796 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:38.998644 16796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:38.998816 16796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:46:38.999970 16796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "a06c052eb5164a1fbe888ddb95310da8"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "72092896176afe52739d68d1e1037453"
server_key_iv: "59467b9a9b10273f4d2768507606ee98"
server_key_version: "encryptionkey@0"
I20260924 17:46:39.000368 16796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "a06c052eb5164a1fbe888ddb95310da8"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "72092896176afe52739d68d1e1037453"
server_key_iv: "59467b9a9b10273f4d2768507606ee98"
server_key_version: "encryptionkey@0"
I20260924 17:46:39.001874 16796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:46:39.002808 16811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:39.003031 16796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:39.003130 16796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "a06c052eb5164a1fbe888ddb95310da8"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "72092896176afe52739d68d1e1037453"
server_key_iv: "59467b9a9b10273f4d2768507606ee98"
server_key_version: "encryptionkey@0"
I20260924 17:46:39.003212 16796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:39.011921 16796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:39.013892 16796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:39.014073 16796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:39.014325 16796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:39.014703 16796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:39.014765 16796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:39.014811 16796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:39.014839 16796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:39.021019 16796 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:34089
I20260924 17:46:39.021080 16924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:34089 every 8 connection(s)
I20260924 17:46:39.021446 16796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:46:39.026340 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16796
I20260924 17:46:39.026496 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:46:39.026711 1058 external_mini_cluster.cc:1428] Setting key 582302bc3d40d47859b742fbcb295e79
Sep 24 17:46:39 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790271998, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:39.031992 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.022342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:59959 (local address 127.1.8.190:40109)
0924 17:46:39.022470 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.022472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.022682 (+ 210us) server_negotiation.cc:407] Connection header received
0924 17:46:39.023223 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.023225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.023244 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.023309 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:39.024312 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.024894 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.025925 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.026367 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.026495 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.029090 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:39.029092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:39.029092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:39.029100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:39.030618 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:39.031104 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:39.031104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:39.031105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:39.031159 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:39.031400 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:39.031401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:39.031401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:39.031513 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:39.031565 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.031839 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.031921 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 17:46:39.032155 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.022306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:59959)
0924 17:46:39.022544 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.022583 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:39.023078 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:39.023373 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:39.023376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:39.023992 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:39.024202 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.024207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.024992 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.024993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.025830 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.025832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.026544 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.026545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.026704 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.027443 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:39.027455 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:39.029005 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:39.030698 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:39.030704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:39.030706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:39.031036 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:39.031220 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:39.031221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:39.031221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:39.031327 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:39.031570 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:39.031574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:39.031704 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 17:46:39.031793 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:46:39.031928 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":163,"threads_started":1}
I20260924 17:46:39.032754 16925 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40109
I20260924 17:46:39.032871 16925 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:39.033053 16925 heartbeater.cc:511] Master 127.1.8.190:40109 requested a full tablet report, sending...
I20260924 17:46:39.033492 16466 ts_manager.cc:194] Registered new tserver with Master: a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089)
I20260924 17:46:39.033851 16466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:59959
I20260924 17:46:39.038643 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:46:39.047712 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.040266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.1.8.190:40109)
0924 17:46:39.040545 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.040548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.040556 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:46:39.040582 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.040582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.040593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.040644 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:39.041565 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.042317 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.043537 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.043750 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.043826 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.045028 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:39.045030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:39.045030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:39.045037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:39.046682 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:39.047009 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:39.047010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:39.047010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:39.047069 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:39.047227 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:39.047227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:39.047227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:39.047373 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:39.047446 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.047542 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.047604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260924 17:46:39.049542 16466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:46:39.051468 16466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:46:39.058382 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.054063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:46635 (local address 127.0.0.1:48920)
0924 17:46:39.054532 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.054542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:39.054598 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:39.054885 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:39.054886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:39.054889 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:39.054956 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.054958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.055913 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.055915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.057135 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.057139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.058031 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.058032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.058117 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.058132 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:46:39.058212 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:46:39.058266 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":8,"thread_start_us":102,"threads_started":1}
I20260924 17:46:39.058419 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.054166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.1.8.130:46635)
0924 17:46:39.054545 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.054551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.054588 (+ 37us) server_negotiation.cc:407] Connection header received
0924 17:46:39.054677 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.054678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.054691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.054790 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:39.055042 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.055799 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.057224 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.057930 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.058009 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.058072 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.058249 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.058302 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":88,"threads_started":1}
I20260924 17:46:39.058652 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.054080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34089 (local address 127.0.0.1:34816)
0924 17:46:39.054683 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.054692 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:39.054727 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:39.055023 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:39.055024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:39.055026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:39.055062 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.055063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.056415 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.056416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.057415 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.057417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.058340 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.058341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.058464 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.058470 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:46:39.058506 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 17:46:39.058562 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":92,"threads_started":1}
I20260924 17:46:39.058992 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.054357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.1.8.131:34089)
0924 17:46:39.054845 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.054850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.054870 (+ 20us) server_negotiation.cc:407] Connection header received
0924 17:46:39.054904 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.054906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.054921 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.055136 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:39.055202 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.056299 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.057498 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.058239 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.058756 (+ 517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.058831 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.058860 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.058905 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":147,"threads_started":1}
I20260924 17:46:39.059518 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.054063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42907 (local address 127.0.0.1:60498)
0924 17:46:39.054257 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.054304 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:39.054379 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:39.054854 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:39.054857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:39.054859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:39.054951 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.054954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.057488 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.057488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.058275 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.058277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.059246 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.059247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.059328 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.059332 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:39.059391 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:46:39.059440 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260924 17:46:39.059552 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.054382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.1.8.129:42907)
0924 17:46:39.054632 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.054637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.054648 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:46:39.054693 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.054695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.054737 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.055032 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:39.055108 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.056287 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.058380 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.059127 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.059236 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.059304 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.059422 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.059464 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":108,"threads_started":1}
I20260924 17:46:39.059950 16859 tablet_service.cc:1467] Processing CreateTablet for tablet 6842702aa76d49c0a8655cdebca225af (DEFAULT_TABLE table=test-table [id=2b128e19e50d4997a02d68a056c31699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:39.059957 16725 tablet_service.cc:1467] Processing CreateTablet for tablet 6842702aa76d49c0a8655cdebca225af (DEFAULT_TABLE table=test-table [id=2b128e19e50d4997a02d68a056c31699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:39.060241 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6842702aa76d49c0a8655cdebca225af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:39.060241 16725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6842702aa76d49c0a8655cdebca225af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:39.063161 16942 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Bootstrap starting.
I20260924 17:46:39.063189 16591 tablet_service.cc:1467] Processing CreateTablet for tablet 6842702aa76d49c0a8655cdebca225af (DEFAULT_TABLE table=test-table [id=2b128e19e50d4997a02d68a056c31699]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:39.063380 16943 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Bootstrap starting.
I20260924 17:46:39.063560 16591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6842702aa76d49c0a8655cdebca225af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:39.064015 16942 tablet_bootstrap.cc:654] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:39.064030 16943 tablet_bootstrap.cc:654] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:39.064522 16942 log.cc:824] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:39.064630 16943 log.cc:824] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:39.065302 16942 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: No bootstrap required, opened a new log
I20260924 17:46:39.065402 16942 ts_tablet_manager.cc:1397] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:39.065435 16943 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: No bootstrap required, opened a new log
I20260924 17:46:39.065517 16943 ts_tablet_manager.cc:1397] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:39.067165 16946 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap starting.
I20260924 17:46:39.067123 16942 raft_consensus.cc:348] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.067323 16942 raft_consensus.cc:374] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:39.067373 16942 raft_consensus.cc:729] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06c052eb5164a1fbe888ddb95310da8, State: Initialized, Role: FOLLOWER
I20260924 17:46:39.067503 16942 consensus_queue.cc:260] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.067804 16942 ts_tablet_manager.cc:1428] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:46:39.067703 16943 raft_consensus.cc:348] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.067881 16943 raft_consensus.cc:374] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:39.067950 16943 raft_consensus.cc:729] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b7362a03734b45a9d4fdf0871ca7e0, State: Initialized, Role: FOLLOWER
I20260924 17:46:39.067972 16925 heartbeater.cc:503] Master 127.1.8.190:40109 was elected leader, sending a full tablet report...
I20260924 17:46:39.068073 16943 consensus_queue.cc:260] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.068192 16946 tablet_bootstrap.cc:654] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:39.068413 16943 ts_tablet_manager.cc:1428] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:46:39.068593 16791 heartbeater.cc:503] Master 127.1.8.190:40109 was elected leader, sending a full tablet report...
I20260924 17:46:39.068854 16946 log.cc:824] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:39.069722 16946 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: No bootstrap required, opened a new log
I20260924 17:46:39.069792 16946 ts_tablet_manager.cc:1397] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:46:39.071430 16946 raft_consensus.cc:348] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.071573 16946 raft_consensus.cc:374] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:39.071595 16946 raft_consensus.cc:729] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ee02c9421d4e2e9d871aeb87bf8fe2, State: Initialized, Role: FOLLOWER
I20260924 17:46:39.071676 16946 consensus_queue.cc:260] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.071943 16946 ts_tablet_manager.cc:1428] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:46:39.071960 16657 heartbeater.cc:503] Master 127.1.8.190:40109 was elected leader, sending a full tablet report...
W20260924 17:46:39.116528 16792 tablet.cc:2369] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:46:39.157241 16950 raft_consensus.cc:482] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:39.157395 16950 raft_consensus.cc:504] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.157998 16950 leader_election.cc:288] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635), a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089)
I20260924 17:46:39.161262 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.158354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49311 (local address 127.1.8.131:34089)
0924 17:46:39.158443 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.158445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.158587 (+ 142us) server_negotiation.cc:407] Connection header received
0924 17:46:39.158667 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.158669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.158681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.158765 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:39.158956 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.159439 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.160450 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.160842 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.160949 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.160967 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.161137 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.161177 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 17:46:39.161311 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.158292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:60579 (local address 127.1.8.130:46635)
0924 17:46:39.158345 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.158347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.158361 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:46:39.158503 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.158505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.158520 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.158637 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:39.158849 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.159369 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.160461 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.160877 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.160994 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.161005 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.161194 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.161236 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:46:39.161310 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.158234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34089 (local address 127.1.8.129:49311)
0924 17:46:39.158507 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.158516 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:39.158607 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:39.158780 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:39.158781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:39.158783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:39.158866 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.158868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.159541 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.159542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.160322 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.160325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.160955 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.160956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.161048 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.161052 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:39.161130 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 17:46:39.161177 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":146,"threads_started":1}
I20260924 17:46:39.161324 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.158213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:46635 (local address 127.1.8.129:60579)
0924 17:46:39.158294 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.158307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:39.158370 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:39.158649 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:39.158650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:39.158652 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:39.158747 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.158749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.159501 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.159503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.160322 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:39.160325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.160994 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.160995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:39.161095 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.161099 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:39.161188 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:46:39.161237 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3}
I20260924 17:46:39.161770 16879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6842702aa76d49c0a8655cdebca225af" candidate_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06c052eb5164a1fbe888ddb95310da8" is_pre_election: true
I20260924 17:46:39.161962 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6842702aa76d49c0a8655cdebca225af" candidate_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" is_pre_election: true
I20260924 17:46:39.162015 16879 raft_consensus.cc:2454] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ee02c9421d4e2e9d871aeb87bf8fe2 in term 0.
I20260924 17:46:39.162094 16745 raft_consensus.cc:2454] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ee02c9421d4e2e9d871aeb87bf8fe2 in term 0.
I20260924 17:46:39.162294 16545 leader_election.cc:302] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a06c052eb5164a1fbe888ddb95310da8, c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters:
I20260924 17:46:39.162513 16950 raft_consensus.cc:2781] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:46:39.162575 16950 raft_consensus.cc:482] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:46:39.162592 16950 raft_consensus.cc:3037] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:39.163430 16950 raft_consensus.cc:504] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.163595 16950 leader_election.cc:288] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 1 election: Requested vote from peers 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635), a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089)
I20260924 17:46:39.163815 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6842702aa76d49c0a8655cdebca225af" candidate_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99b7362a03734b45a9d4fdf0871ca7e0"
I20260924 17:46:39.163801 16879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6842702aa76d49c0a8655cdebca225af" candidate_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06c052eb5164a1fbe888ddb95310da8"
I20260924 17:46:39.163914 16745 raft_consensus.cc:3037] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:39.163944 16879 raft_consensus.cc:3037] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:39.164848 16879 raft_consensus.cc:2454] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ee02c9421d4e2e9d871aeb87bf8fe2 in term 1.
I20260924 17:46:39.164848 16745 raft_consensus.cc:2454] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ee02c9421d4e2e9d871aeb87bf8fe2 in term 1.
I20260924 17:46:39.165055 16545 leader_election.cc:302] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a06c052eb5164a1fbe888ddb95310da8, c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters:
I20260924 17:46:39.165189 16950 raft_consensus.cc:2781] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:39.165359 16950 raft_consensus.cc:686] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 LEADER]: Becoming Leader. State: Replica: c3ee02c9421d4e2e9d871aeb87bf8fe2, State: Running, Role: LEADER
I20260924 17:46:39.165477 16950 consensus_queue.cc:237] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:39.166517 16466 catalog_manager.cc:5495] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } health_report { overall_health: UNKNOWN } } }
I20260924 17:46:39.175079 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.172754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.1.8.129:42907)
0924 17:46:39.172815 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:39.172817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:39.172825 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:46:39.172885 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:39.172886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:39.172912 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:39.172975 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:39.173096 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.173573 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.174309 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:39.174558 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:39.174612 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:39.174760 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:39.174867 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:39.174928 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:39.174940 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 17:46:39.174958 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 17:46:39.177208 16879 raft_consensus.cc:1261] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Refusing update from remote peer c3ee02c9421d4e2e9d871aeb87bf8fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:39.177208 16745 raft_consensus.cc:1261] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Refusing update from remote peer c3ee02c9421d4e2e9d871aeb87bf8fe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:39.177611 16952 consensus_queue.cc:1035] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:39.177757 16950 consensus_queue.cc:1035] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:39.182940 16465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:39.183075 16465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:39.183882 16465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Sending DeleteTablet for 3 replicas of tablet 6842702aa76d49c0a8655cdebca225af
I20260924 17:46:39.184233 16725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6842702aa76d49c0a8655cdebca225af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:39 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:48920
I20260924 17:46:39.184247 16591 tablet_service.cc:1514] Processing DeleteTablet for tablet 6842702aa76d49c0a8655cdebca225af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:39 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:60498
I20260924 17:46:39.184489 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16528
I20260924 17:46:39.184492 16859 tablet_service.cc:1514] Processing DeleteTablet for tablet 6842702aa76d49c0a8655cdebca225af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:39 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:34816
I20260924 17:46:39.184692 16966 tablet_replica.cc:330] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: stopping tablet replica
I20260924 17:46:39.184792 16966 raft_consensus.cc:2229] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:39.184909 16966 raft_consensus.cc:2258] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:39.185143 16964 tablet_replica.cc:330] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: stopping tablet replica
I20260924 17:46:39.185241 16964 raft_consensus.cc:2229] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:39.185343 16964 raft_consensus.cc:2258] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:39.185503 16966 ts_tablet_manager.cc:1905] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:39.185807 16964 ts_tablet_manager.cc:1905] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:39.187469 16966 ts_tablet_manager.cc:1918] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:39.187566 16966 log.cc:1196] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/6842702aa76d49c0a8655cdebca225af
I20260924 17:46:39.187810 16964 ts_tablet_manager.cc:1918] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:39.187904 16964 log.cc:1196] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/6842702aa76d49c0a8655cdebca225af
I20260924 17:46:39.187955 16966 ts_tablet_manager.cc:1939] T 6842702aa76d49c0a8655cdebca225af P a06c052eb5164a1fbe888ddb95310da8: Deleting consensus metadata
I20260924 17:46:39.188192 16964 ts_tablet_manager.cc:1939] T 6842702aa76d49c0a8655cdebca225af P 99b7362a03734b45a9d4fdf0871ca7e0: Deleting consensus metadata
I20260924 17:46:39.188421 16451 catalog_manager.cc:4841] TS a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089): tablet 6842702aa76d49c0a8655cdebca225af (table test-table [id=2b128e19e50d4997a02d68a056c31699]) successfully deleted
I20260924 17:46:39.189007 16450 catalog_manager.cc:4841] TS 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635): tablet 6842702aa76d49c0a8655cdebca225af (table test-table [id=2b128e19e50d4997a02d68a056c31699]) successfully deleted
W20260924 17:46:39.191679 16453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:42907 (error 108)
I20260924 17:46:39.192345 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16662
I20260924 17:46:39.193277 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:39.193107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42907 (local address 127.0.0.1:60514)
0924 17:46:39.193177 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:39.193223 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 17:46:39.193439 16453 catalog_manager.cc:4584] TS c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6842702aa76d49c0a8655cdebca225af: Network error: Client connection negotiation failed: client connection to 127.1.8.129:42907: connect: Connection refused (error 111)
I20260924 17:46:39.197715 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16796
I20260924 17:46:39.203228 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16434
I20260924 17:46:59.209457 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40109
--webserver_interface=127.1.8.190
--webserver_port=34889
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:46:59.293083 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:59.293313 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:59.293337 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:59.294597 16968 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 17:46:59.294641 16968 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 17:46:59.294654 16968 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 17:46:59.294667 16968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:46:59.294713 16968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:59.294745 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:59.294763 16968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:46:59.294788 16968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:46:59.296396 16968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:40109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:59.296684 16968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:59.296981 16968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:59.306500 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:59.306623 16968 server_base.cc:1034] running on GCE node
W20260924 17:46:59.306509 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:59.306802 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:59.307156 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:59.307508 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:59.308686 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790272019308686 us; error 65 us; skew 500 ppm
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:59.311468 16968 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:46:59.312002 16968 webserver.cc:460] Webserver started at http://127.1.8.190:34889/ using document root <none> and password file <none>
I20260924 17:46:59.312273 16968 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:59.312368 16968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:59.313822 16968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:46:59.314603 16984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.314783 16968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:59.314890 16968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "ffc84deea31c475ba7d5cccaba11ab04"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "324e25868a082628e46b5b39abc1acad"
server_key_iv: "43afa7c60bd46555372fdd182404e95e"
server_key_version: "encryptionkey@0"
I20260924 17:46:59.315366 16968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:59.353343 16968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:59.355965 16968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:59.356148 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:59.360303 16968 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:40109
I20260924 17:46:59.360369 17036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:40109 every 8 connection(s)
I20260924 17:46:59.360791 16968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:46:59.362372 17037 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:46:59.363152 17037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap starting.
I20260924 17:46:59.365490 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 16968
I20260924 17:46:59.365799 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:42907
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:40109
--webserver_port=39785
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:59.371773 17037 log.cc:824] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:59.373924 17037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:59.374238 17037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap complete.
I20260924 17:46:59.376639 17037 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:59.376896 17037 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc84deea31c475ba7d5cccaba11ab04, State: Initialized, Role: FOLLOWER
I20260924 17:46:59.377094 17037 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:59.377173 17037 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:46:59.377231 17037 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:46:59.377298 17037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 1 FOLLOWER]: Advancing to term 2
I20260924 17:46:59.378350 17037 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:59.378494 17037 leader_election.cc:302] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc84deea31c475ba7d5cccaba11ab04; no voters:
I20260924 17:46:59.378715 17037 leader_election.cc:288] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 17:46:59.378751 17041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 17:46:59.378912 17041 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 LEADER]: Becoming Leader. State: Replica: ffc84deea31c475ba7d5cccaba11ab04, State: Running, Role: LEADER
I20260924 17:46:59.378997 17037 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:46:59.379053 17041 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:46:59.379336 17041 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc84deea31c475ba7d5cccaba11ab04. Latest consensus state: current_term: 2 leader_uuid: "ffc84deea31c475ba7d5cccaba11ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } } }
I20260924 17:46:59.379443 17041 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:59.379336 17042 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ffc84deea31c475ba7d5cccaba11ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } } }
I20260924 17:46:59.379664 17045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:46:59.380570 17045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:46:59.380735 17045 catalog_manager.cc:1259] Loaded cluster ID: d6a41c748cbe44b4914ae35a76bd662c
I20260924 17:46:59.381084 17045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:46:59.380767 17042 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:46:59.382566 17045 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:46:59.382766 17045 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Loaded TSK: 0
I20260924 17:46:59.392596 17045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Generated new TSK 1
I20260924 17:46:59.392726 17045 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:46:59.454619 17039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:59.454854 17039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:59.454879 17039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:59.456143 17039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:59.456202 17039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:59.456231 17039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:46:59.457782 17039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:42907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=39785
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:59.458063 17039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:59.458336 17039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:59.463596 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:59.463642 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:59.463838 17039 server_base.cc:1034] running on GCE node
W20260924 17:46:59.463650 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:59.464083 17039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:59.464398 17039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:59.465557 17039 hybrid_clock.cc:648] HybridClock initialized: now 1790272019465541 us; error 32 us; skew 500 ppm
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:59.468216 17039 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:46:59.468647 17039 webserver.cc:460] Webserver started at http://127.1.8.129:39785/ using document root <none> and password file <none>
I20260924 17:46:59.468892 17039 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:59.468972 17039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:59.470471 17039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:46:59.471239 17074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.471428 17039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:59.471532 17039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "bc232f210a132d0767346f9609a62191"
server_key_iv: "d0a7f5fc986127d52d1bf0cb130a1541"
server_key_version: "encryptionkey@0"
I20260924 17:46:59.471917 17039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:59.489360 17039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:59.491901 17039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:59.492069 17039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:59.492316 17039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:59.492761 17081 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:46:59.494690 17039 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:46:59.494760 17039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:46:59.494815 17039 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:46:59.495497 17039 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:46:59.495555 17039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:46:59.495633 17081 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap starting.
I20260924 17:46:59.503469 17039 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42907
I20260924 17:46:59.503876 17039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:46:59.507232 17081 log.cc:824] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:59.507295 17188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42907 every 8 connection(s)
I20260924 17:46:59.510151 17081 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:46:59.510603 17081 tablet_bootstrap.cc:492] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap complete.
I20260924 17:46:59.511299 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17039
I20260924 17:46:59.511582 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:46635
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:40109
--webserver_port=35233
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:59.511868 17081 ts_tablet_manager.cc:1397] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260924 17:46:59.513374 17081 raft_consensus.cc:348] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:59.513880 17081 raft_consensus.cc:729] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ee02c9421d4e2e9d871aeb87bf8fe2, State: Initialized, Role: FOLLOWER
I20260924 17:46:59.514051 17081 consensus_queue.cc:260] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } } peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } }
I20260924 17:46:59.514386 17081 ts_tablet_manager.cc:1428] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:59.519869 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.504594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:49393)
0924 17:46:59.506598 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.506641 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.507109 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.508216 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.508220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.508691 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:59.508955 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.508962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.510691 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.510693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.512155 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.512158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.513134 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.513136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.513267 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.514009 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:59.514031 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:59.515771 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:59.517910 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:59.517919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:59.517921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:59.518212 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:59.518584 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:59.518585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:59.518585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:59.518690 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:59.519169 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:59.519175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:59.519340 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.519459 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.519653 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1957,"thread_start_us":104,"threads_started":1}
I20260924 17:46:59.519868 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.504641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49393 (local address 127.1.8.190:40109)
0924 17:46:59.507649 (+ 3008us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.507660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.507716 (+ 56us) server_negotiation.cc:407] Connection header received
0924 17:46:59.507893 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.507904 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.507978 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.508164 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:59.509122 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.510556 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.512270 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.512892 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.513015 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.515891 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:59.515896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:59.515896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:59.515916 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:59.517803 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.518312 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.518315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.518316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.518462 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.518766 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.518767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.518768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.519093 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:59.519189 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.519520 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.519653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2942,"thread_start_us":85,"threads_started":1}
I20260924 17:46:59.520586 17189 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40109
I20260924 17:46:59.520701 17189 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:59.520893 17189 heartbeater.cc:511] Master 127.1.8.190:40109 requested a full tablet report, sending...
I20260924 17:46:59.521624 17001 ts_manager.cc:194] Registered new tserver with Master: c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907)
I20260924 17:46:59.522852 17001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:49393
I20260924 17:46:59.523753 17189 heartbeater.cc:503] Master 127.1.8.190:40109 was elected leader, sending a full tablet report...
I20260924 17:46:59.534013 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.522510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.1.8.129:42907)
0924 17:46:59.530262 (+ 7752us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.530267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.530281 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:46:59.530318 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.530320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.530335 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.530513 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:59.530804 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.531479 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.533121 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.533607 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.533671 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.533727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.533910 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.533945 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7711,"thread_start_us":66,"threads_started":1}
I20260924 17:46:59.534039 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.522282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42907 (local address 127.0.0.1:58518)
0924 17:46:59.522501 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.522547 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.522635 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.530599 (+ 7964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.530603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.530608 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:59.530708 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.530711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.531621 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.531622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.533016 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.533020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.533701 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.533702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.533793 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.533808 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.533885 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.533947 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":321,"threads_started":1}
I20260924 17:46:59.534616 17117 tablet_service.cc:1514] Processing DeleteTablet for tablet 6842702aa76d49c0a8655cdebca225af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:39 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:58518
I20260924 17:46:59.535202 17198 tablet_replica.cc:330] stopping tablet replica
I20260924 17:46:59.535324 17198 raft_consensus.cc:2229] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:59.535419 17198 raft_consensus.cc:2258] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:59.535743 17198 ts_tablet_manager.cc:1905] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:59.538352 17198 ts_tablet_manager.cc:1918] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:59.538514 17198 log.cc:1196] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal/wals/6842702aa76d49c0a8655cdebca225af
I20260924 17:46:59.538811 17198 ts_tablet_manager.cc:1939] T 6842702aa76d49c0a8655cdebca225af P c3ee02c9421d4e2e9d871aeb87bf8fe2: Deleting consensus metadata
I20260924 17:46:59.539376 16988 catalog_manager.cc:4841] TS c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907): tablet 6842702aa76d49c0a8655cdebca225af (table test-table [id=2b128e19e50d4997a02d68a056c31699]) successfully deleted
W20260924 17:46:59.603531 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:59.603768 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:59.603796 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:59.604926 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:59.604967 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:59.604995 17194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:46:59.606462 17194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:46635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=35233
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:59.606678 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:59.606940 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:46:59.612430 17194 server_base.cc:1034] running on GCE node
W20260924 17:46:59.612329 17206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:59.612329 17203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:59.612339 17204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:59.612785 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:59.613078 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:59.614253 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790272019614244 us; error 36 us; skew 500 ppm
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:59.617020 17194 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:46:59.617555 17194 webserver.cc:460] Webserver started at http://127.1.8.130:35233/ using document root <none> and password file <none>
I20260924 17:46:59.617830 17194 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:59.617923 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:59.619443 17194 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:59.620075 17213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.620275 17194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:59.620362 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "99b7362a03734b45a9d4fdf0871ca7e0"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "d815f79803aa3e9b964c697052c4fabc"
server_key_iv: "d291251926619ba57a015aef566509a6"
server_key_version: "encryptionkey@0"
I20260924 17:46:59.620749 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:59.638149 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:59.640709 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:59.640889 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:59.641144 17194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:59.641474 17194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:59.641528 17194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.641599 17194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:59.641647 17194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.648187 17194 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:46635
I20260924 17:46:59.648238 17326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:46635 every 8 connection(s)
I20260924 17:46:59.648627 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:59.658684 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17194
I20260924 17:46:59.659030 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:34089
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:40109
--webserver_port=39303
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:46:59.659202 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.649394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33193 (local address 127.1.8.190:40109)
0924 17:46:59.649451 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.649454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.649779 (+ 325us) server_negotiation.cc:407] Connection header received
0924 17:46:59.650268 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.650269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.650284 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.650340 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:59.651321 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.651853 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.652856 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.653077 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.653145 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.656124 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:59.656125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:59.656126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:59.656134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:59.657604 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.658016 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.658016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.658017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.658075 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.658318 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.658318 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.658318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.658467 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:59.658531 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.658960 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.659057 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:46:59.659423 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.649394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:33193)
0924 17:46:59.649589 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.649679 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.650169 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.650422 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.650425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.651009 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:59.651211 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.651215 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.651944 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.651945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.652754 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.652755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.653163 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.653164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.653269 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.654227 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:59.654236 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:59.656006 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:59.657689 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:59.657696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:59.657698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:59.657942 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:59.658139 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:59.658140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:59.658140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:59.658254 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:59.658535 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:59.658543 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:59.658742 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.658924 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.659123 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260924 17:46:59.659957 17327 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40109
I20260924 17:46:59.660099 17327 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:59.660321 17327 heartbeater.cc:511] Master 127.1.8.190:40109 requested a full tablet report, sending...
I20260924 17:46:59.660879 17001 ts_manager.cc:194] Registered new tserver with Master: 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635)
I20260924 17:46:59.661271 17001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:33193
W20260924 17:46:59.745265 17330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:46:59.745510 17330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:46:59.745553 17330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:46:59.746816 17330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:46:59.746881 17330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:46:59.746928 17330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:46:59.748456 17330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:34089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=39303
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:46:59.748747 17330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:46:59.748991 17330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:46:59.754164 17336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:59.754169 17338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:46:59.754169 17335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:46:59.754359 17330 server_base.cc:1034] running on GCE node
I20260924 17:46:59.754683 17330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:46:59.754957 17330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:46:59.756124 17330 hybrid_clock.cc:648] HybridClock initialized: now 1790272019756105 us; error 39 us; skew 500 ppm
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:46:59.758780 17330 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:46:59.759347 17330 webserver.cc:460] Webserver started at http://127.1.8.131:39303/ using document root <none> and password file <none>
I20260924 17:46:59.759629 17330 fs_manager.cc:359] Metadata directory not provided
I20260924 17:46:59.759727 17330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:46:59.761245 17330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:59.761881 17345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.762056 17330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:59.762156 17330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "a06c052eb5164a1fbe888ddb95310da8"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "72092896176afe52739d68d1e1037453"
server_key_iv: "59467b9a9b10273f4d2768507606ee98"
server_key_version: "encryptionkey@0"
I20260924 17:46:59.762581 17330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:46:59.780182 17330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:46:59.782850 17330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:46:59.783070 17330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:46:59.783363 17330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:46:59.783710 17330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:46:59.783767 17330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.783811 17330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:46:59.783836 17330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:46:59.789882 17330 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:34089
I20260924 17:46:59.789929 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:34089 every 8 connection(s)
I20260924 17:46:59.790307 17330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:46:59.794972 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17330
Sep 24 17:46:59 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:46:59.800689 17192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.791195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:51389 (local address 127.1.8.190:40109)
0924 17:46:59.791247 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.791249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.791500 (+ 251us) server_negotiation.cc:407] Connection header received
0924 17:46:59.791993 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.791994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.792004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.792074 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:59.793147 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.793646 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.794693 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.794884 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.794950 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.797364 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:59.797365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:59.797365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:59.797371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:59.798869 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.799362 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.799363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.799363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.799411 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.799740 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.799741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.799741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.799920 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:59.799997 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.800484 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.800576 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:46:59.800976 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.791179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:51389)
0924 17:46:59.791359 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.791403 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.791908 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.792157 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.792161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.792827 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:46:59.793030 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.793035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.793732 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.793733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.794569 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.794571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.794982 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.794983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.795154 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.795763 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:46:59.795772 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:46:59.797277 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:46:59.798951 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:59.798958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:59.798961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:59.799285 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:59.799475 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:46:59.799476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:46:59.799476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:46:59.799639 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:46:59.800181 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:46:59.800186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:46:59.800317 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.800444 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.800686 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260924 17:46:59.801561 17459 heartbeater.cc:347] Connected to a master server at 127.1.8.190:40109
I20260924 17:46:59.801673 17459 heartbeater.cc:464] Registering TS with master...
I20260924 17:46:59.801855 17459 heartbeater.cc:511] Master 127.1.8.190:40109 requested a full tablet report, sending...
I20260924 17:46:59.802335 17001 ts_manager.cc:194] Registered new tserver with Master: a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089)
I20260924 17:46:59.802497 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.795411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.1.8.190:40109)
0924 17:46:59.795552 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.795554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.795720 (+ 166us) server_negotiation.cc:407] Connection header received
0924 17:46:59.795797 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.795798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.795811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.795893 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:46:59.796544 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.797091 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.797917 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.798119 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.798206 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.799663 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:46:59.799664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:46:59.799664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:46:59.799670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:46:59.801219 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.801606 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.801606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.801607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.801687 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:46:59.801904 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:46:59.801905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:46:59.801906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:46:59.802036 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:46:59.802239 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.802280 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.802361 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260924 17:46:59.802834 17001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:51389
I20260924 17:46:59.803287 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:46:59.805836 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.803697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.1.8.190:40109)
0924 17:46:59.803736 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.803738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.803801 (+ 63us) server_negotiation.cc:407] Connection header received
0924 17:46:59.803864 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.803865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.803892 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.803952 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:59.804038 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.804484 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.805218 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.805396 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.805444 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.805571 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:59.805664 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:59.805727 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.805759 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.805780 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:46:59.806121 17000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:46:59.807758 17000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:46:59.809993 17117 tablet_service.cc:1467] Processing CreateTablet for tablet 9a313967d1a141568dec9ece09fd7c23 (DEFAULT_TABLE table=test-table [id=94519ecc78264c3f958677d1174eabe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:59.810199 17117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a313967d1a141568dec9ece09fd7c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:59.811899 17081 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap starting.
I20260924 17:46:59.812217 17081 tablet_bootstrap.cc:654] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:59.812947 17081 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: No bootstrap required, opened a new log
I20260924 17:46:59.812999 17081 ts_tablet_manager.cc:1397] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 17:46:59.813104 17081 raft_consensus.cc:348] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.813177 17081 raft_consensus.cc:374] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:59.813230 17081 raft_consensus.cc:729] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ee02c9421d4e2e9d871aeb87bf8fe2, State: Initialized, Role: FOLLOWER
I20260924 17:46:59.813282 17081 consensus_queue.cc:260] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.813382 17081 ts_tablet_manager.cc:1428] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 17:46:59.813704 17196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.809684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34089 (local address 127.0.0.1:44074)
0924 17:46:59.809722 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.809734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.809772 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.810258 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.810259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.810261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:59.810309 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.810310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.811515 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.811516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.812489 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.812492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.813446 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.813447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.813540 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.813545 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.813581 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.813631 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 17:46:59.814030 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.809873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.1.8.131:34089)
0924 17:46:59.810084 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.810088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.810099 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:46:59.810139 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.810141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.810156 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.810261 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:59.810366 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.811431 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.812580 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.813344 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.813805 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.813874 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.813903 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.813946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":102,"threads_started":1}
I20260924 17:46:59.814559 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.809981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.1.8.130:46635)
0924 17:46:59.810236 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.810241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.810633 (+ 392us) server_negotiation.cc:407] Connection header received
0924 17:46:59.810713 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.810715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.810731 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.810803 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:59.811013 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.812082 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.813370 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.814135 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.814228 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.814286 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.814434 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.814482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":77,"threads_started":1}
I20260924 17:46:59.814759 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.809689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:46635 (local address 127.0.0.1:53284)
0924 17:46:59.810566 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.810577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.810638 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.810892 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.810893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.810894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:59.810931 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.810932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.812196 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.812197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.813290 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.813293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.814238 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.814239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.814303 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.814307 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.814637 (+ 330us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.814686 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":12,"thread_start_us":114,"threads_started":1}
I20260924 17:46:59.814731 17393 tablet_service.cc:1467] Processing CreateTablet for tablet 9a313967d1a141568dec9ece09fd7c23 (DEFAULT_TABLE table=test-table [id=94519ecc78264c3f958677d1174eabe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:59.815126 17393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a313967d1a141568dec9ece09fd7c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:59.817837 17469 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Bootstrap starting.
I20260924 17:46:59.818351 17261 tablet_service.cc:1467] Processing CreateTablet for tablet 9a313967d1a141568dec9ece09fd7c23 (DEFAULT_TABLE table=test-table [id=94519ecc78264c3f958677d1174eabe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:46:59.818629 17469 tablet_bootstrap.cc:654] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:59.818750 17261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a313967d1a141568dec9ece09fd7c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:46:59.819238 17469 log.cc:824] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:59.820010 17469 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: No bootstrap required, opened a new log
I20260924 17:46:59.820098 17469 ts_tablet_manager.cc:1397] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:59.821599 17469 raft_consensus.cc:348] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.821795 17469 raft_consensus.cc:374] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:59.821848 17469 raft_consensus.cc:729] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06c052eb5164a1fbe888ddb95310da8, State: Initialized, Role: FOLLOWER
I20260924 17:46:59.821847 17471 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Bootstrap starting.
I20260924 17:46:59.821981 17469 consensus_queue.cc:260] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.822309 17459 heartbeater.cc:503] Master 127.1.8.190:40109 was elected leader, sending a full tablet report...
I20260924 17:46:59.822315 17469 ts_tablet_manager.cc:1428] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:59.822564 17471 tablet_bootstrap.cc:654] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Neither blocks nor log segments found. Creating new log.
I20260924 17:46:59.823249 17471 log.cc:824] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Log is configured to *not* fsync() on all Append() calls
I20260924 17:46:59.823939 17471 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: No bootstrap required, opened a new log
I20260924 17:46:59.823995 17471 ts_tablet_manager.cc:1397] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:59.825361 17471 raft_consensus.cc:348] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.825510 17471 raft_consensus.cc:374] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:46:59.825533 17471 raft_consensus.cc:729] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b7362a03734b45a9d4fdf0871ca7e0, State: Initialized, Role: FOLLOWER
I20260924 17:46:59.825623 17471 consensus_queue.cc:260] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.825912 17471 ts_tablet_manager.cc:1428] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:46:59.825927 17327 heartbeater.cc:503] Master 127.1.8.190:40109 was elected leader, sending a full tablet report...
I20260924 17:46:59.869051 17474 raft_consensus.cc:482] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:46:59.869174 17474 raft_consensus.cc:504] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.869524 17474 leader_election.cc:288] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089), c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907)
I20260924 17:46:59.872603 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.869725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34089 (local address 127.1.8.130:37247)
0924 17:46:59.869928 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.869938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.869997 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.870157 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.870158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.870159 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:59.870249 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.870251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.870857 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.870858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.871671 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.871674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.872261 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.872262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.872350 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.872356 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.872430 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.872481 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260924 17:46:59.872589 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.869817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:37247 (local address 127.1.8.131:34089)
0924 17:46:59.869865 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.869867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.870006 (+ 139us) server_negotiation.cc:407] Connection header received
0924 17:46:59.870053 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.870054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.870069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.870123 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:59.870318 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.870781 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.871797 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.872177 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.872233 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.872243 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.872474 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.872517 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:46:59.873028 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.869706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42907 (local address 127.1.8.130:47939)
0924 17:46:59.869776 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.869789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:46:59.869834 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:46:59.870010 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:46:59.870011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:46:59.870014 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:46:59.870090 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.870101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.870885 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.870886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.871673 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:46:59.871675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.872802 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.872803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:46:59.872876 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.872879 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:46:59.872929 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 17:46:59.872967 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":47}
I20260924 17:46:59.872972 17413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a313967d1a141568dec9ece09fd7c23" candidate_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06c052eb5164a1fbe888ddb95310da8" is_pre_election: true
I20260924 17:46:59.873062 17197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.869819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:47939 (local address 127.1.8.129:42907)
0924 17:46:59.869865 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.869867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.869876 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:46:59.869907 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.869908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.869920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.869993 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:46:59.870178 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.870784 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.871772 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.872688 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.872759 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.872770 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.872953 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.873006 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:46:59.873144 17413 raft_consensus.cc:2454] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99b7362a03734b45a9d4fdf0871ca7e0 in term 0.
I20260924 17:46:59.873382 17215 leader_election.cc:302] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99b7362a03734b45a9d4fdf0871ca7e0, a06c052eb5164a1fbe888ddb95310da8; no voters:
I20260924 17:46:59.873371 17143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a313967d1a141568dec9ece09fd7c23" candidate_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" is_pre_election: true
I20260924 17:46:59.873535 17143 raft_consensus.cc:2454] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99b7362a03734b45a9d4fdf0871ca7e0 in term 0.
I20260924 17:46:59.873528 17474 raft_consensus.cc:2781] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:46:59.873602 17474 raft_consensus.cc:482] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:46:59.873647 17474 raft_consensus.cc:3037] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:59.874586 17474 raft_consensus.cc:504] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.874733 17474 leader_election.cc:288] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [CANDIDATE]: Term 1 election: Requested vote from peers a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089), c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907)
I20260924 17:46:59.874884 17143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a313967d1a141568dec9ece09fd7c23" candidate_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2"
I20260924 17:46:59.874950 17413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a313967d1a141568dec9ece09fd7c23" candidate_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06c052eb5164a1fbe888ddb95310da8"
I20260924 17:46:59.874976 17143 raft_consensus.cc:3037] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:59.875041 17413 raft_consensus.cc:3037] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:46:59.875765 17143 raft_consensus.cc:2454] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b7362a03734b45a9d4fdf0871ca7e0 in term 1.
I20260924 17:46:59.875803 17413 raft_consensus.cc:2454] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b7362a03734b45a9d4fdf0871ca7e0 in term 1.
I20260924 17:46:59.875963 17217 leader_election.cc:302] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99b7362a03734b45a9d4fdf0871ca7e0, c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters:
I20260924 17:46:59.876051 17474 raft_consensus.cc:2781] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:46:59.876221 17474 raft_consensus.cc:686] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 LEADER]: Becoming Leader. State: Replica: 99b7362a03734b45a9d4fdf0871ca7e0, State: Running, Role: LEADER
I20260924 17:46:59.876297 17474 consensus_queue.cc:237] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:46:59.877198 17000 catalog_manager.cc:5495] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } health_report { overall_health: UNKNOWN } } }
I20260924 17:46:59.888507 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.885902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.1.8.130:46635)
0924 17:46:59.885974 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:46:59.885976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:46:59.885987 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:46:59.886054 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:46:59.886055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:46:59.886079 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:46:59.886171 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:46:59.886312 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.886860 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.887655 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:46:59.887871 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:46:59.887950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:46:59.888125 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:46:59.888244 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:46:59.888344 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 17:46:59.888364 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 17:46:59.888385 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:46:59.890448 17413 raft_consensus.cc:1261] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Refusing update from remote peer 99b7362a03734b45a9d4fdf0871ca7e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:59.890448 17143 raft_consensus.cc:1261] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Refusing update from remote peer 99b7362a03734b45a9d4fdf0871ca7e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:46:59.890859 17476 consensus_queue.cc:1035] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:59.891083 17474 consensus_queue.cc:1035] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:46:59.893405 17479 mvcc.cc:205] Tried to move back new op lower bound from 7332954193468252160 to 7332954193414041600. Current Snapshot: MvccSnapshot[applied={T|T < 7332954193468252160}]
I20260924 17:46:59.893918 17481 mvcc.cc:205] Tried to move back new op lower bound from 7332954193468252160 to 7332954193414041600. Current Snapshot: MvccSnapshot[applied={T|T < 7332954193468252160}]
I20260924 17:46:59.896377 17001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:59.896483 17001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:46:59.897353 17001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Sending DeleteTablet for 3 replicas of tablet 9a313967d1a141568dec9ece09fd7c23
I20260924 17:46:59.897748 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 17039
I20260924 17:46:59.897760 17261 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a313967d1a141568dec9ece09fd7c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:59 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:53284
I20260924 17:46:59.897830 17117 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a313967d1a141568dec9ece09fd7c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:59 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:58518
I20260924 17:46:59.897921 17198 tablet_replica.cc:330] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: stopping tablet replica
I20260924 17:46:59.897961 17198 raft_consensus.cc:2229] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:59.898021 17198 raft_consensus.cc:2258] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:59.898542 17489 tablet_replica.cc:330] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: stopping tablet replica
I20260924 17:46:59.898584 17393 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a313967d1a141568dec9ece09fd7c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:46:59 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:44074
I20260924 17:46:59.898651 17489 raft_consensus.cc:2229] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:46:59.898746 17489 raft_consensus.cc:2258] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:59.898782 17490 tablet_replica.cc:330] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: stopping tablet replica
I20260924 17:46:59.898876 17490 raft_consensus.cc:2229] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:46:59.898973 17490 raft_consensus.cc:2258] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:46:59.899242 17489 ts_tablet_manager.cc:1905] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:59.899437 17490 ts_tablet_manager.cc:1905] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:46:59.901649 17490 ts_tablet_manager.cc:1918] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:59.901655 17489 ts_tablet_manager.cc:1918] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:46:59.901737 17489 log.cc:1196] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/9a313967d1a141568dec9ece09fd7c23
I20260924 17:46:59.901737 17490 log.cc:1196] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/9a313967d1a141568dec9ece09fd7c23
I20260924 17:46:59.902073 17490 ts_tablet_manager.cc:1939] T 9a313967d1a141568dec9ece09fd7c23 P a06c052eb5164a1fbe888ddb95310da8: Deleting consensus metadata
I20260924 17:46:59.902072 17489 ts_tablet_manager.cc:1939] T 9a313967d1a141568dec9ece09fd7c23 P 99b7362a03734b45a9d4fdf0871ca7e0: Deleting consensus metadata
I20260924 17:46:59.902559 16985 catalog_manager.cc:4841] TS 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635): tablet 9a313967d1a141568dec9ece09fd7c23 (table test-table [id=94519ecc78264c3f958677d1174eabe1]) successfully deleted
I20260924 17:46:59.902688 16986 catalog_manager.cc:4841] TS a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089): tablet 9a313967d1a141568dec9ece09fd7c23 (table test-table [id=94519ecc78264c3f958677d1174eabe1]) successfully deleted
W20260924 17:46:59.905105 16988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:42907 (error 108)
I20260924 17:46:59.905435 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 17194
I20260924 17:46:59.906116 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:46:59.905559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:42907 (local address 127.0.0.1:58520)
0924 17:46:59.906044 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 17:46:59.906081 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461}
W20260924 17:46:59.907063 16988 catalog_manager.cc:4584] TS c3ee02c9421d4e2e9d871aeb87bf8fe2 (127.1.8.129:42907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a313967d1a141568dec9ece09fd7c23: Network error: Client connection negotiation failed: client connection to 127.1.8.129:42907: connect: Connection refused (error 111)
I20260924 17:46:59.910377 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 17330
I20260924 17:46:59.914949 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 16968
I20260924 17:47:19.920714 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40109
--webserver_interface=127.1.8.190
--webserver_port=34889
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:40109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:20.005357 17492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:20.005659 17492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:20.005704 17492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:20.007033 17492 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 17:47:20.007079 17492 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 17:47:20.007093 17492 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 17:47:20.007105 17492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:20.007119 17492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:20.007133 17492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:20.007145 17492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:20.007164 17492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:20.008656 17492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:40109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:40109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=34889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:20.008877 17492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:20.009058 17492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:20.014767 17497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.014868 17500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.014878 17498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:20.014832 17492 server_base.cc:1034] running on GCE node
I20260924 17:47:20.015267 17492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:20.015594 17492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:20.016762 17492 hybrid_clock.cc:648] HybridClock initialized: now 1790272040016748 us; error 35 us; skew 500 ppm
Sep 24 17:47:20 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:20.019357 17492 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:47:20.019868 17492 webserver.cc:460] Webserver started at http://127.1.8.190:34889/ using document root <none> and password file <none>
I20260924 17:47:20.020098 17492 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:20.020169 17492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:20.021559 17492 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:20.022246 17507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.022445 17492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:47:20.022575 17492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "ffc84deea31c475ba7d5cccaba11ab04"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "324e25868a082628e46b5b39abc1acad"
server_key_iv: "43afa7c60bd46555372fdd182404e95e"
server_key_version: "encryptionkey@0"
I20260924 17:47:20.022940 17492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:20.033056 17492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:20.035120 17492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:20.035288 17492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:20.039757 17492 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:40109
I20260924 17:47:20.039825 17559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:40109 every 8 connection(s)
I20260924 17:47:20.040241 17492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:20.041780 17560 sys_catalog.cc:263] Verifying existing consensus state
I20260924 17:47:20.042546 17560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap starting.
I20260924 17:47:20.046377 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17492
I20260924 17:47:20.046705 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:42907
--local_ip_for_outbound_sockets=127.1.8.129
--tserver_master_addrs=127.1.8.190:40109
--webserver_port=39785
--webserver_interface=127.1.8.129
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:47:20.051404 17560 log.cc:824] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:20.053752 17560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:47:20.054101 17560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Bootstrap complete.
I20260924 17:47:20.056490 17560 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:47:20.056749 17560 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc84deea31c475ba7d5cccaba11ab04, State: Initialized, Role: FOLLOWER
I20260924 17:47:20.056944 17560 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:47:20.057026 17560 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:20.057077 17560 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:20.057127 17560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 2 FOLLOWER]: Advancing to term 3
I20260924 17:47:20.058145 17560 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:47:20.058269 17560 leader_election.cc:302] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc84deea31c475ba7d5cccaba11ab04; no voters:
I20260924 17:47:20.058490 17560 leader_election.cc:288] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 17:47:20.058528 17564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 17:47:20.058658 17564 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [term 3 LEADER]: Becoming Leader. State: Replica: ffc84deea31c475ba7d5cccaba11ab04, State: Running, Role: LEADER
I20260924 17:47:20.058786 17560 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:20.058809 17564 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } }
I20260924 17:47:20.059125 17564 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc84deea31c475ba7d5cccaba11ab04. Latest consensus state: current_term: 3 leader_uuid: "ffc84deea31c475ba7d5cccaba11ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } } }
I20260924 17:47:20.059209 17564 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:20.059130 17565 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ffc84deea31c475ba7d5cccaba11ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc84deea31c475ba7d5cccaba11ab04" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 40109 } } }
I20260924 17:47:20.059368 17565 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:20.059434 17570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:20.060333 17570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:20.060508 17570 catalog_manager.cc:1259] Loaded cluster ID: d6a41c748cbe44b4914ae35a76bd662c
I20260924 17:47:20.060706 17570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:20.061570 17570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:20.061780 17570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Loaded TSK: 0
I20260924 17:47:20.061842 17570 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffc84deea31c475ba7d5cccaba11ab04: Loaded TSK: 1
F20260924 17:47:20.062089 17570 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f13397ee9ad google::LogMessage::Fail() at ??:0
@ 0x7f13397f188a google::LogMessage::SendToLog() at ??:0
@ 0x7f13397ee4de google::LogMessage::Flush() at ??:0
@ 0x7f13397efa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f133bf5b104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f133bf58d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f13417481cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f134174848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1341748594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f134170ec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1341718ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f133b6bc0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f133b6b1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f133c5156db start_thread at ??:0
@ 0x7f13382ec61f clone at ??:0
W20260924 17:47:20.166858 17562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:20.167156 17562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:20.167199 17562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:20.169082 17562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:20.169152 17562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:20.169196 17562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:20.171499 17562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:42907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=39785
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:20.172003 17562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:20.172335 17562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:20.178028 17602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.178042 17604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.178274 17601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:20.178584 17562 server_base.cc:1034] running on GCE node
I20260924 17:47:20.178824 17562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:20.179108 17562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:20.180296 17562 hybrid_clock.cc:648] HybridClock initialized: now 1790272040180280 us; error 33 us; skew 500 ppm
Sep 24 17:47:20 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:20.183039 17562 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:47:20.183542 17562 webserver.cc:460] Webserver started at http://127.1.8.129:39785/ using document root <none> and password file <none>
I20260924 17:47:20.183791 17562 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:20.183864 17562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:20.185282 17562 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:20.186035 17611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.186232 17562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:20.186339 17562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "bc232f210a132d0767346f9609a62191"
server_key_iv: "d0a7f5fc986127d52d1bf0cb130a1541"
server_key_version: "encryptionkey@0"
I20260924 17:47:20.186761 17562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:20.194474 17562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:20.196833 17562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:20.197001 17562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:20.197234 17562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:20.197677 17618 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 17:47:20.199738 17562 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 17:47:20.199801 17562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 17:47:20.199838 17562 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 17:47:20.200784 17562 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 17:47:20.200829 17562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 17:47:20.200920 17618 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap starting.
I20260924 17:47:20.208642 17562 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:42907
I20260924 17:47:20.208809 17725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:42907 every 8 connection(s)
I20260924 17:47:20.209205 17562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:47:20.212426 17618 log.cc:824] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:20.213526 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17562
I20260924 17:47:20.213850 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:46635
--local_ip_for_outbound_sockets=127.1.8.130
--tserver_master_addrs=127.1.8.190:40109
--webserver_port=35233
--webserver_interface=127.1.8.130
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:47:20.215006 17618 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 17:47:20.215358 17618 tablet_bootstrap.cc:492] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Bootstrap complete.
I20260924 17:47:20.216570 17618 ts_tablet_manager.cc:1397] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20260924 17:47:20.217628 17618 raft_consensus.cc:348] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:47:20.217865 17618 raft_consensus.cc:729] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ee02c9421d4e2e9d871aeb87bf8fe2, State: Initialized, Role: FOLLOWER
I20260924 17:47:20.218037 17618 consensus_queue.cc:260] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:47:20.218369 17618 ts_tablet_manager.cc:1428] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260924 17:47:20.302356 17728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:47:20.209921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:50431)
0924 17:47:20.210144 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.210210 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:20.210974 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:20.302129 (+ 91155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
W20260924 17:47:20.302687 17726 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:40109 (0 consecutive failures): Network error: Failed to ping master at 127.1.8.190:40109: Client connection negotiation failed: client connection to 127.1.8.190:40109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:47:20.303061 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.302943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:59491)
0924 17:47:20.302999 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.303042 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:47:20.303377 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.303293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:49327)
0924 17:47:20.303339 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.303363 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:47:20.303493 17726 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 17:47:20.318984 17730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:20.319244 17730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:20.319304 17730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:20.320549 17730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:20.320629 17730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:20.320676 17730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:20.322301 17730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:46635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=35233
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:20.322618 17730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:20.322861 17730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:20.328258 17736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.328258 17739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.328258 17737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:20.328596 17730 server_base.cc:1034] running on GCE node
I20260924 17:47:20.328780 17730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:20.329005 17730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:20.330192 17730 hybrid_clock.cc:648] HybridClock initialized: now 1790272040330176 us; error 30 us; skew 500 ppm
Sep 24 17:47:20 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:20.333119 17730 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:47:20.333680 17730 webserver.cc:460] Webserver started at http://127.1.8.130:35233/ using document root <none> and password file <none>
I20260924 17:47:20.333928 17730 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:20.333979 17730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:20.335551 17730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:20.336241 17746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.336623 17730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:20.336711 17730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "99b7362a03734b45a9d4fdf0871ca7e0"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "d815f79803aa3e9b964c697052c4fabc"
server_key_iv: "d291251926619ba57a015aef566509a6"
server_key_version: "encryptionkey@0"
I20260924 17:47:20.337141 17730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:20.387867 17730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:20.390491 17730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:20.390673 17730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:20.390897 17730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:20.391211 17730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:20.391244 17730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.391304 17730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:20.391335 17730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.397142 17730 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:46635
I20260924 17:47:20.397186 17859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:46635 every 8 connection(s)
I20260924 17:47:20.397539 17730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:20.398746 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.398215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:50209)
0924 17:47:20.398492 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.398599 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
W20260924 17:47:20.399207 17860 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:40109 (0 consecutive failures): Network error: Failed to ping master at 127.1.8.190:40109: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
I20260924 17:47:20.399631 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.399534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:58421)
0924 17:47:20.399582 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.399611 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:47:20.399974 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.399842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:57689)
0924 17:47:20.399907 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.399936 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 17:47:20.400128 17860 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:47:20.402383 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17730
I20260924 17:47:20.402788 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:34089
--local_ip_for_outbound_sockets=127.1.8.131
--tserver_master_addrs=127.1.8.190:40109
--webserver_port=39303
--webserver_interface=127.1.8.131
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:42283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:20.485735 17863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:20.485961 17863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:20.485985 17863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:20.487223 17863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:20.487277 17863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:20.487344 17863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:20.488914 17863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:42283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:34089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=39303
--tserver_master_addrs=127.1.8.190:40109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:20.489180 17863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:20.489423 17863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:47:20.494797 17863 server_base.cc:1034] running on GCE node
W20260924 17:47:20.494764 17868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.494791 17869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:20.494768 17871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:20.495249 17863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:20.495477 17863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:20.496642 17863 hybrid_clock.cc:648] HybridClock initialized: now 1790272040496618 us; error 41 us; skew 500 ppm
Sep 24 17:47:20 dist-test-slave-r7hf krb5kdc[16421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:20.499244 17863 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:47:20.499811 17863 webserver.cc:460] Webserver started at http://127.1.8.131:39303/ using document root <none> and password file <none>
I20260924 17:47:20.500052 17863 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:20.500098 17863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:20.501605 17863 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:20.502295 17878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.502506 17863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:20.502614 17863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "a06c052eb5164a1fbe888ddb95310da8"
format_stamp: "Formatted at 2026-09-24 17:46:38 on dist-test-slave-r7hf"
server_key: "72092896176afe52739d68d1e1037453"
server_key_iv: "59467b9a9b10273f4d2768507606ee98"
server_key_version: "encryptionkey@0"
I20260924 17:47:20.502982 17863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:20.528822 17863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:20.531409 17863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:20.531576 17863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:20.531828 17863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:20.532141 17863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:20.532193 17863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.532234 17863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:20.532259 17863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:20.538244 17863 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:34089
I20260924 17:47:20.538286 17991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:34089 every 8 connection(s)
I20260924 17:47:20.538702 17863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:47:20.539858 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.539488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:36753)
0924 17:47:20.539639 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.539714 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
W20260924 17:47:20.540282 17992 heartbeater.cc:644] Failed to heartbeat to 127.1.8.190:40109 (0 consecutive failures): Network error: Failed to ping master at 127.1.8.190:40109: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
I20260924 17:47:20.540640 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.540558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:57221)
0924 17:47:20.540595 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.540622 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 17:47:20.540932 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:20.540847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:48257)
0924 17:47:20.540892 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:20.540919 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:47:20.541061 17992 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 17:47:20.548170 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 17863
I20260924 17:47:21.304014 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.303795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:48665)
0924 17:47:21.303922 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:21.303950 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260924 17:47:21.400676 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.400446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:41971)
0924 17:47:21.400588 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:21.400624 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":57,"threads_started":1}
I20260924 17:47:21.438711 17998 raft_consensus.cc:482] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:21.438843 17998 raft_consensus.cc:504] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:47:21.439265 17998 leader_election.cc:288] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089), 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635)
Sep 24 17:47:21 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.131@KRBTEST.COM
Sep 24 17:47:21 dist-test-slave-r7hf krb5kdc[16421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.130@KRBTEST.COM
I20260924 17:47:21.450193 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.439618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:56233 (local address 127.1.8.130:46635)
0924 17:47:21.439765 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:21.439778 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:21.439813 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:47:21.440416 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:21.440419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:21.440475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:21.440632 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:21.441470 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.442350 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.443470 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.443873 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.443999 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:21.446743 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:21.446747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:21.446747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:21.446759 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:21.448511 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:21.448931 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:21.448933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:21.448934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:21.449021 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:21.449312 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:21.449313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:21.449313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:21.449547 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:21.449632 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:21.449938 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 17:47:21.450066 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":52,"threads_started":1}
I20260924 17:47:21.450210 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.439604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:58333 (local address 127.1.8.131:34089)
0924 17:47:21.439762 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:21.439792 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:21.439815 (+ 23us) server_negotiation.cc:407] Connection header received
0924 17:47:21.440415 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:21.440417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:21.440474 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:21.440709 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:21.441463 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.442356 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.443472 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.443876 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.443999 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:21.446462 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:21.446469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:21.446469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:21.446487 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:21.448387 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:21.448882 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:21.448886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:21.448886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:21.449020 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:21.449289 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:21.449291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:21.449291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:21.449548 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:21.449655 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:21.449944 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 17:47:21.450101 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260924 17:47:21.450294 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.439470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:46635 (local address 127.1.8.129:56233)
0924 17:47:21.439540 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:21.439554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:21.439631 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:21.440660 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:21.440662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:21.441144 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:21.441335 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:21.441340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.442537 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.442538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:21.443373 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:21.443376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.443972 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.443973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:21.444103 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:21.444784 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:21.444789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:21.446636 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:21.448599 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:21.448599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:21.448600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:21.448851 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:21.449097 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:21.449098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:21.449099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:21.449225 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:21.449624 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:21.449631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:21.449767 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 17:47:21.449872 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 17:47:21.450163 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":170}
I20260924 17:47:21.450177 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.439471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34089 (local address 127.1.8.129:58333)
0924 17:47:21.439679 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:21.439688 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:21.439745 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:21.440713 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:21.440714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:21.441144 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:21.441335 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:21.441340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.442545 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.442546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:21.443374 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:21.443376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:21.443973 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:21.443974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:21.444103 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:21.444728 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:21.444736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:21.446309 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:21.448481 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:21.448488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:21.448491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:21.448803 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:21.449095 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:21.449096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:21.449096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:21.449222 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:21.449655 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:21.449656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:21.449765 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 17:47:21.449872 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 17:47:21.450079 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":89,"thread_start_us":101,"threads_started":1}
W20260924 17:47:21.451283 17613 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
W20260924 17:47:21.451555 17612 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
I20260924 17:47:21.451596 17612 leader_election.cc:302] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters: 99b7362a03734b45a9d4fdf0871ca7e0, a06c052eb5164a1fbe888ddb95310da8
I20260924 17:47:21.451740 17998 raft_consensus.cc:2726] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:47:21.541723 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:21.541492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:48039)
0924 17:47:21.541632 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:21.541682 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 17:47:22.304759 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:22.304537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:40165)
0924 17:47:22.304670 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:22.304712 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260924 17:47:22.401444 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:22.401156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:48831)
0924 17:47:22.401344 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:22.401396 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":63,"threads_started":1}
I20260924 17:47:22.542503 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:22.542213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:49805)
0924 17:47:22.542340 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:22.542381 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 17:47:23.296217 18006 raft_consensus.cc:482] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:23.296417 18006 raft_consensus.cc:504] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:47:23.296815 18006 leader_election.cc:288] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089), 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635)
W20260924 17:47:23.297204 17612 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
W20260924 17:47:23.297298 17613 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
I20260924 17:47:23.297406 17613 leader_election.cc:302] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters: 99b7362a03734b45a9d4fdf0871ca7e0, a06c052eb5164a1fbe888ddb95310da8
I20260924 17:47:23.297484 18006 raft_consensus.cc:2726] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:47:23.305689 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:23.305376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:49231)
0924 17:47:23.305591 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:23.305639 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":121,"threads_started":1}
I20260924 17:47:23.402252 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:23.401991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:44229)
0924 17:47:23.402152 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:23.402201 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260924 17:47:23.543136 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:23.542913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:59915)
0924 17:47:23.543039 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:23.543091 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 17:47:24.306658 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:24.306308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:42757)
0924 17:47:24.306568 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:24.306612 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":212,"threads_started":1}
I20260924 17:47:24.402902 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:24.402688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:57515)
0924 17:47:24.402811 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:24.402855 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260924 17:47:24.543828 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:24.543605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:46659)
0924 17:47:24.543740 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:24.543781 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260924 17:47:25.021463 18013 raft_consensus.cc:482] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:25.021603 18013 raft_consensus.cc:504] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:47:25.021817 18013 leader_election.cc:288] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089), 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635)
W20260924 17:47:25.022181 17613 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
W20260924 17:47:25.022256 17612 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
I20260924 17:47:25.022320 17612 leader_election.cc:302] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters: 99b7362a03734b45a9d4fdf0871ca7e0, a06c052eb5164a1fbe888ddb95310da8
I20260924 17:47:25.022432 18013 raft_consensus.cc:2726] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:47:25.307562 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:25.307300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:50427)
0924 17:47:25.307474 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:25.307515 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260924 17:47:25.403793 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:25.403537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:48859)
0924 17:47:25.403697 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:25.403747 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260924 17:47:25.544596 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:25.544312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:37253)
0924 17:47:25.544503 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:25.544550 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260924 17:47:26.308368 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:26.308100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:48107)
0924 17:47:26.308244 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:26.308318 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260924 17:47:26.404515 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:26.404306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:38611)
0924 17:47:26.404422 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:26.404471 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 17:47:26.545356 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:26.545107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:40683)
0924 17:47:26.545269 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:26.545309 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":76,"threads_started":1}
I20260924 17:47:27.309144 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:27.308913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:56815)
0924 17:47:27.309044 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:27.309086 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260924 17:47:27.405222 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:27.404989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:41005)
0924 17:47:27.405129 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:27.405175 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260924 17:47:27.546136 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:27.545896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:35691)
0924 17:47:27.546046 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:27.546091 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260924 17:47:28.309996 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:28.309700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:54257)
0924 17:47:28.309881 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:28.309929 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":62,"threads_started":1}
I20260924 17:47:28.405845 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:28.405644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:58207)
0924 17:47:28.405767 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:28.405800 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260924 17:47:28.546900 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:28.546673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:57175)
0924 17:47:28.546808 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:28.546852 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260924 17:47:29.310848 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:29.310587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:43141)
0924 17:47:29.310744 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:29.310791 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260924 17:47:29.406675 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:29.406450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:47635)
0924 17:47:29.406585 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:29.406629 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260924 17:47:29.547766 18028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:29.547510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:39919)
0924 17:47:29.547670 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:29.547719 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260924 17:47:29.839072 18029 raft_consensus.cc:482] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:29.839191 18029 raft_consensus.cc:504] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06c052eb5164a1fbe888ddb95310da8" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34089 } } peers { permanent_uuid: "99b7362a03734b45a9d4fdf0871ca7e0" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 46635 } } peers { permanent_uuid: "c3ee02c9421d4e2e9d871aeb87bf8fe2" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 42907 } }
I20260924 17:47:29.839387 18029 leader_election.cc:288] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089), 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635)
W20260924 17:47:29.839845 17612 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 99b7362a03734b45a9d4fdf0871ca7e0 (127.1.8.130:46635): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
W20260924 17:47:29.839928 17613 leader_election.cc:341] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a06c052eb5164a1fbe888ddb95310da8 (127.1.8.131:34089): Not found: Tablet not found: 9a313967d1a141568dec9ece09fd7c23
I20260924 17:47:29.839962 17613 leader_election.cc:302] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3ee02c9421d4e2e9d871aeb87bf8fe2; no voters: 99b7362a03734b45a9d4fdf0871ca7e0, a06c052eb5164a1fbe888ddb95310da8
I20260924 17:47:29.840050 18029 raft_consensus.cc:2726] T 9a313967d1a141568dec9ece09fd7c23 P c3ee02c9421d4e2e9d871aeb87bf8fe2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 17:47:30.311672 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:30.311424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:42495)
0924 17:47:30.311580 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:30.311623 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260924 17:47:30.407378 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:30.407165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:45341)
0924 17:47:30.407288 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:30.407333 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 17:47:30.548542 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:30.548306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:59707)
0924 17:47:30.548458 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:30.548495 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260924 17:47:31.312464 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:31.312231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:48279)
0924 17:47:31.312376 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:31.312417 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260924 17:47:31.408339 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:31.408050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:33855)
0924 17:47:31.408246 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:31.408292 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":133,"threads_started":1}
I20260924 17:47:31.549320 18035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:31.549083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:47971)
0924 17:47:31.549229 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:31.549273 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260924 17:47:32.313225 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:32.312991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:49313)
0924 17:47:32.313133 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:32.313178 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260924 17:47:32.409067 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:32.408839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:52909)
0924 17:47:32.408972 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:32.409021 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260924 17:47:32.550081 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:32.549851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:48595)
0924 17:47:32.549988 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:32.550033 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260924 17:47:33.314101 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:33.313845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:43543)
0924 17:47:33.314003 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:33.314053 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260924 17:47:33.409974 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:33.409696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:56863)
0924 17:47:33.409885 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:33.409930 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260924 17:47:33.551016 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:33.550725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:43467)
0924 17:47:33.550897 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:33.550954 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260924 17:47:34.314976 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:34.314736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:58291)
0924 17:47:34.314884 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:34.314929 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260924 17:47:34.410708 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:34.410503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:56461)
0924 17:47:34.410625 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:34.410662 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260924 17:47:34.551841 18044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:34.551553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:48439)
0924 17:47:34.551686 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:34.551760 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260924 17:47:35.315845 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:35.315588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.129:39481)
0924 17:47:35.315755 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:35.315798 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260924 17:47:35.411574 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:35.411342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.130:44049)
0924 17:47:35.411484 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:35.411527 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260924 17:47:35.552824 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:35.552584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:40109 (local address 127.1.8.131:56765)
0924 17:47:35.552722 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:35.552768 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.190:40109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260924 17:47:35.552958 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 17562
I20260924 17:47:35.557592 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 17730
I20260924 17:47:35.562040 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 17863
2026-09-24T17:47:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58452 ms)
[----------] 30 tests from SecurityITest (264175 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:35 dist-test-slave-r7hf krb5kdc[18050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:35 dist-test-slave-r7hf krb5kdc[18050](info): set up 2 sockets
Sep 24 17:47:35 dist-test-slave-r7hf krb5kdc[18050](info): commencing operation
krb5kdc: starting...
W20260924 17:47:36.626598 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:36 dist-test-slave-r7hf krb5kdc[18050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:47:36.754745 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:35501
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:35501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:36.839567 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:36.839818 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:36.839848 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:36.841127 18063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:36.841192 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:36.841209 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:36.841296 18063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:36.841318 18063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:36.842965 18063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:35501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:35501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:36.843274 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:36.843564 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:36.850276 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:36.850253 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:36.850343 18063 server_base.cc:1034] running on GCE node
W20260924 17:47:36.850244 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:36.850730 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:36.851055 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:36.852201 18063 hybrid_clock.cc:648] HybridClock initialized: now 1790272056852169 us; error 49 us; skew 500 ppm
Sep 24 17:47:36 dist-test-slave-r7hf krb5kdc[18050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272056, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:36.854871 18063 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:47:36.855288 18063 webserver.cc:460] Webserver started at http://127.1.8.190:40345/ using document root <none> and password file <none>
I20260924 17:47:36.855549 18063 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:36.855597 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:36.855733 18063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:36.856838 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "c57e93fecc4040f9ac3454c1b9f70d1d"
format_stamp: "Formatted at 2026-09-24 17:47:36 on dist-test-slave-r7hf"
server_key: "7386b9044f2935a88aa438ac1f96b27b"
server_key_iv: "d5086eb11d9913b8eb112172e3d0c53a"
server_key_version: "encryptionkey@0"
I20260924 17:47:36.857216 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "c57e93fecc4040f9ac3454c1b9f70d1d"
format_stamp: "Formatted at 2026-09-24 17:47:36 on dist-test-slave-r7hf"
server_key: "7386b9044f2935a88aa438ac1f96b27b"
server_key_iv: "d5086eb11d9913b8eb112172e3d0c53a"
server_key_version: "encryptionkey@0"
I20260924 17:47:36.858644 18063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:47:36.859373 18078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:36.859578 18063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:47:36.859656 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "c57e93fecc4040f9ac3454c1b9f70d1d"
format_stamp: "Formatted at 2026-09-24 17:47:36 on dist-test-slave-r7hf"
server_key: "7386b9044f2935a88aa438ac1f96b27b"
server_key_iv: "d5086eb11d9913b8eb112172e3d0c53a"
server_key_version: "encryptionkey@0"
I20260924 17:47:36.859715 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:36.869076 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:36.871261 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:36.871424 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:36.875317 18063 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:35501
I20260924 17:47:36.875362 18130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:35501 every 8 connection(s)
I20260924 17:47:36.875775 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:36.876689 18131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:36.879107 18131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d: Bootstrap starting.
I20260924 17:47:36.879702 18131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:36.880048 18131 log.cc:824] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:36.880180 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18063
I20260924 17:47:36.880296 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:36.880491 1058 external_mini_cluster.cc:1428] Setting key 59ac932e65031f82a08e128635bc9851
I20260924 17:47:36.880950 18131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d: No bootstrap required, opened a new log
I20260924 17:47:36.882921 18131 raft_consensus.cc:348] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 35501 } }
I20260924 17:47:36.883061 18131 raft_consensus.cc:374] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:36.883085 18131 raft_consensus.cc:729] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57e93fecc4040f9ac3454c1b9f70d1d, State: Initialized, Role: FOLLOWER
I20260924 17:47:36.883220 18131 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 35501 } }
I20260924 17:47:36.883319 18131 raft_consensus.cc:388] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:36.883364 18131 raft_consensus.cc:482] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:36.883411 18131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:36.883975 18131 raft_consensus.cc:504] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 35501 } }
I20260924 17:47:36.884059 18131 leader_election.cc:302] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57e93fecc4040f9ac3454c1b9f70d1d; no voters:
I20260924 17:47:36.884261 18131 leader_election.cc:288] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:36.884294 18136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:36.884428 18136 raft_consensus.cc:686] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [term 1 LEADER]: Becoming Leader. State: Replica: c57e93fecc4040f9ac3454c1b9f70d1d, State: Running, Role: LEADER
I20260924 17:47:36.884522 18131 sys_catalog.cc:564] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:36.884541 18136 consensus_queue.cc:237] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 35501 } }
I20260924 17:47:36.885041 18137 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c57e93fecc4040f9ac3454c1b9f70d1d. Latest consensus state: current_term: 1 leader_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 35501 } } }
I20260924 17:47:36.885175 18137 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:36.885686 18138 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57e93fecc4040f9ac3454c1b9f70d1d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 35501 } } }
I20260924 17:47:36.885805 18138 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:36.885968 18145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:36.887176 18145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:47:36 dist-test-slave-r7hf krb5kdc[18050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:36.888700 18145 catalog_manager.cc:1347] Generated new cluster ID: 90744077a163415e811b609624d07444
I20260924 17:47:36.888768 18145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:36.892835 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:36.881033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.1.8.190:35501)
0924 17:47:36.881230 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:36.881240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:36.881332 (+ 92us) server_negotiation.cc:407] Connection header received
0924 17:47:36.882126 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:36.882134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:36.882185 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:36.882340 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:36.883003 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:36.884549 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:36.885660 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:36.886305 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:36.886448 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:36.889093 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:36.889097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:36.889097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:36.889109 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:36.891128 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:36.891580 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:36.891582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:36.891583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:36.891685 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:36.891901 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:36.891902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:36.891902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:36.892193 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:36.892348 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:36.892479 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 17:47:36.892610 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260924 17:47:36.900877 18145 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:36.901429 18145 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:36.905316 18145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d: Generated new TSK 0
I20260924 17:47:36.905536 18145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:47:36.934288 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:35501
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:37.018355 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:37.018658 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:37.018710 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:37.019973 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:37.020053 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:37.020102 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:37.021641 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:37.021963 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:37.022217 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:37.027554 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:37.027539 18162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:37.027539 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.028054 18157 server_base.cc:1034] running on GCE node
I20260924 17:47:37.028291 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:37.028553 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:37.029732 18157 hybrid_clock.cc:648] HybridClock initialized: now 1790272057029718 us; error 30 us; skew 500 ppm
Sep 24 17:47:37 dist-test-slave-r7hf krb5kdc[18050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:37.032392 18157 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:47:37.032930 18157 webserver.cc:460] Webserver started at http://127.1.8.129:43113/ using document root <none> and password file <none>
I20260924 17:47:37.033190 18157 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:37.033263 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:37.033422 18157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:37.034617 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "b18f2fb5d1724f85a19e6e57120f8655"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "d208ec4c1e3894b869ca467fd63aeb5a"
server_key_iv: "42c9120d37676b19f8b8a564ae5a061a"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.034991 18157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "b18f2fb5d1724f85a19e6e57120f8655"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "d208ec4c1e3894b869ca467fd63aeb5a"
server_key_iv: "42c9120d37676b19f8b8a564ae5a061a"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.036430 18157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:47:37.037392 18172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.037640 18157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:37.037750 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "b18f2fb5d1724f85a19e6e57120f8655"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "d208ec4c1e3894b869ca467fd63aeb5a"
server_key_iv: "42c9120d37676b19f8b8a564ae5a061a"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.037846 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:37.051487 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:37.054021 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:37.054235 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:37.054545 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:37.054891 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:37.054949 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.054991 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:37.055018 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.060894 18157 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:33623
I20260924 17:47:37.060968 18285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:33623 every 8 connection(s)
I20260924 17:47:37.061313 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
Sep 24 17:47:37 dist-test-slave-r7hf krb5kdc[18050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:37.069871 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18157
I20260924 17:47:37.069980 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:47:37.070226 1058 external_mini_cluster.cc:1428] Setting key f822c6663412be9243e06c55fc10c170
I20260924 17:47:37.071476 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.062175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:47907 (local address 127.1.8.190:35501)
0924 17:47:37.062269 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.062271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.062571 (+ 300us) server_negotiation.cc:407] Connection header received
0924 17:47:37.063059 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.063059 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.063070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.063115 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:37.064199 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.064725 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.065698 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.065919 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.065984 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.068191 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:37.068193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:37.068194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:37.068200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:37.069683 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.070191 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.070192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.070192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.070245 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.070626 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.070627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.070627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.070766 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:37.070822 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.071225 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.071335 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 17:47:37.071481 18288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.062156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:35501 (local address 127.1.8.129:47907)
0924 17:47:37.062426 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.062475 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.062972 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.063199 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.063203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.063806 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:37.064006 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.064011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.064821 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.064821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.065610 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.065611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.066001 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.066002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.066102 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.066686 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:37.066695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:37.068087 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:37.069767 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:37.069774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:37.069776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:37.070112 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:37.070320 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:37.070321 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:37.070322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:37.070510 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:37.070836 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:37.070843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:37.071026 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.071172 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.071312 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":133,"threads_started":1}
I20260924 17:47:37.072135 18286 heartbeater.cc:347] Connected to a master server at 127.1.8.190:35501
I20260924 17:47:37.072257 18286 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:37.072459 18286 heartbeater.cc:511] Master 127.1.8.190:35501 requested a full tablet report, sending...
I20260924 17:47:37.073040 18095 ts_manager.cc:194] Registered new tserver with Master: b18f2fb5d1724f85a19e6e57120f8655 (127.1.8.129:33623)
I20260924 17:47:37.073647 18095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:47907
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:47:37.098820 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:35501
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:37.180948 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:37.181212 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:37.181252 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:37.182499 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:37.182575 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:37.182623 18291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:37.184286 18291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:37.184559 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:37.184840 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:37.190119 18296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:37.190122 18297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.190459 18291 server_base.cc:1034] running on GCE node
W20260924 17:47:37.190122 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.190889 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:37.191201 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:37.192371 18291 hybrid_clock.cc:648] HybridClock initialized: now 1790272057192359 us; error 48 us; skew 500 ppm
Sep 24 17:47:37 dist-test-slave-r7hf krb5kdc[18050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:37.195122 18291 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:47:37.195585 18291 webserver.cc:460] Webserver started at http://127.1.8.130:34231/ using document root <none> and password file <none>
I20260924 17:47:37.195819 18291 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:37.195868 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:37.196033 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:37.197147 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "8b5dfe96647b4f6387a69a0fe80713a6"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "c440e26034ff887503a771e7217be658"
server_key_iv: "9386c65fdcf6ef701fc384196cd3473b"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.197531 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "8b5dfe96647b4f6387a69a0fe80713a6"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "c440e26034ff887503a771e7217be658"
server_key_iv: "9386c65fdcf6ef701fc384196cd3473b"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.198993 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:37.199793 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.199980 18291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:37.200073 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "8b5dfe96647b4f6387a69a0fe80713a6"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "c440e26034ff887503a771e7217be658"
server_key_iv: "9386c65fdcf6ef701fc384196cd3473b"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.200150 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:37.219662 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:37.221793 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:37.222018 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:37.222302 18291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:37.222692 18291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:37.222757 18291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.222805 18291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:37.222836 18291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.229422 18291 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39303
I20260924 17:47:37.229507 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39303 every 8 connection(s)
I20260924 17:47:37.229871 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:37.233942 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18291
I20260924 17:47:37.234027 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:47:37.234246 1058 external_mini_cluster.cc:1428] Setting key ee6ac84a1ed5a25f298d5bcd0b51cc72
Sep 24 17:47:37 dist-test-slave-r7hf krb5kdc[18050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:37.240243 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.230725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:52983 (local address 127.1.8.190:35501)
0924 17:47:37.230797 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.230799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.231010 (+ 211us) server_negotiation.cc:407] Connection header received
0924 17:47:37.231530 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.231531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.231542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.231605 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:37.232731 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.233237 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.234618 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.234811 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.234904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.237170 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:37.237172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:37.237172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:37.237182 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:37.238766 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.239206 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.239207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.239208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.239272 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.239548 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.239549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.239549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.239679 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:37.239760 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.240053 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.240145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:47:37.240460 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.230684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:35501 (local address 127.1.8.130:52983)
0924 17:47:37.230869 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.230913 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.231437 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.231698 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.231702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.232391 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:37.232614 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.232619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.233395 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.233396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.234466 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.234470 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.234910 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.234912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.235057 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.235654 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:37.235663 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:37.237066 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:37.238866 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:37.238873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:37.238875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:37.239127 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:37.239353 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:37.239354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:37.239355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:37.239478 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:37.239751 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:37.239757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:37.239916 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.240033 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.240209 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260924 17:47:37.240933 18420 heartbeater.cc:347] Connected to a master server at 127.1.8.190:35501
I20260924 17:47:37.241034 18420 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:37.241245 18420 heartbeater.cc:511] Master 127.1.8.190:35501 requested a full tablet report, sending...
I20260924 17:47:37.241714 18095 ts_manager.cc:194] Registered new tserver with Master: 8b5dfe96647b4f6387a69a0fe80713a6 (127.1.8.130:39303)
I20260924 17:47:37.242326 18095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:52983
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:47:37.262624 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:35501
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:37.348472 18425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:37.348752 18425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:37.348798 18425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:37.350108 18425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:37.350199 18425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:37.350250 18425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:37.351920 18425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:35501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:37.352218 18425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:37.352463 18425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:37.357764 18431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.357834 18425 server_base.cc:1034] running on GCE node
W20260924 17:47:37.357800 18430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:37.357785 18433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.358186 18425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:37.358548 18425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:37.359709 18425 hybrid_clock.cc:648] HybridClock initialized: now 1790272057359697 us; error 21 us; skew 500 ppm
Sep 24 17:47:37 dist-test-slave-r7hf krb5kdc[18050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:37.362313 18425 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:47:37.362788 18425 webserver.cc:460] Webserver started at http://127.1.8.131:40131/ using document root <none> and password file <none>
I20260924 17:47:37.363293 18425 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:37.363371 18425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:37.363533 18425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:37.364674 18425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "24878f326eb94822abe246148e605fa4"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "440a873672d611996c06ef0c37240815"
server_key_iv: "c47725b4bf30fe9a7848aae52d3c1812"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.365072 18425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "24878f326eb94822abe246148e605fa4"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "440a873672d611996c06ef0c37240815"
server_key_iv: "c47725b4bf30fe9a7848aae52d3c1812"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.366547 18425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:37.367357 18440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.367532 18425 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:37.367637 18425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "24878f326eb94822abe246148e605fa4"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "440a873672d611996c06ef0c37240815"
server_key_iv: "c47725b4bf30fe9a7848aae52d3c1812"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.367722 18425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:37.377395 18425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:37.379967 18425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:37.380148 18425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:37.380410 18425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:37.380779 18425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:37.380837 18425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.380880 18425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:37.380906 18425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.387550 18425 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:45267
I20260924 17:47:37.387593 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:45267 every 8 connection(s)
I20260924 17:47:37.387931 18425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:47:37 dist-test-slave-r7hf krb5kdc[18050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272057, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:37.397092 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.388672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:40427 (local address 127.1.8.190:35501)
0924 17:47:37.388716 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.388718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.388962 (+ 244us) server_negotiation.cc:407] Connection header received
0924 17:47:37.389439 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.389440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.389451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.389505 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:37.390518 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.391015 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.392062 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.392250 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.392307 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.394645 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:37.394646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:37.394647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:37.394654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:37.395872 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.396251 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.396252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.396252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.396304 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.396547 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.396548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.396548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.396655 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:37.396705 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.396956 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.397021 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:47:37.397356 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.388674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:35501 (local address 127.1.8.131:40427)
0924 17:47:37.388830 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.388868 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.389353 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.389564 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.389567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.390156 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:37.390371 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.390376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.391103 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.391104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.391975 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.391977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.392349 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.392350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.392461 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.393063 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:37.393072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:37.394555 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:37.395946 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:37.395952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:37.395954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:37.396180 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:37.396380 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:37.396381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:37.396381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:37.396488 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:37.396708 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:37.396712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:37.396838 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.396926 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.397058 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 17:47:37.397787 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18425
I20260924 17:47:37.397881 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:47:37.397994 18554 heartbeater.cc:347] Connected to a master server at 127.1.8.190:35501
I20260924 17:47:37.398084 1058 external_mini_cluster.cc:1428] Setting key 6e20ad1c58fc3bb3462cc5261d0e223f
I20260924 17:47:37.398118 18554 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:37.398335 18554 heartbeater.cc:511] Master 127.1.8.190:35501 requested a full tablet report, sending...
I20260924 17:47:37.398798 18095 ts_manager.cc:194] Registered new tserver with Master: 24878f326eb94822abe246148e605fa4 (127.1.8.131:45267)
I20260924 17:47:37.399277 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:47:37.399348 18095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:40427
I20260924 17:47:37.406963 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.400679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.1.8.190:35501)
0924 17:47:37.400752 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.400753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.400797 (+ 44us) server_negotiation.cc:407] Connection header received
0924 17:47:37.400852 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.400853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.400861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.400906 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:37.401400 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.401901 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.402722 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.402980 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.403047 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.404374 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:37.404375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:37.404375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:37.404381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:37.405801 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.406153 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.406154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.406154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.406208 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:37.406444 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:37.406445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:37.406445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:37.406599 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:37.406680 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.406774 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.406857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:47:37.408782 18095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:47:37.410498 18095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:47:37.417238 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.412923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33623 (local address 127.0.0.1:43846)
0924 17:47:37.413036 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.413078 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.413166 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.413681 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.413685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.413688 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:37.413811 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.413813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.414829 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.414830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.416025 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.416028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.416927 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.416927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.417007 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.417022 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.417094 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.417154 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":35,"thread_start_us":60,"threads_started":1}
I20260924 17:47:37.417250 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.413297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.1.8.129:33623)
0924 17:47:37.413493 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.413499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.413510 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:47:37.413555 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.413557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.413579 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.413671 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:37.413874 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.414723 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.416114 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.416839 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.416929 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.416991 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.417120 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.417160 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260924 17:47:37.417658 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.413069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.1.8.131:45267)
0924 17:47:37.413515 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.413520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.413528 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:47:37.413561 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.413563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.413576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.413664 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:37.413991 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.415169 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.416370 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.417033 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.417104 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.417157 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.417516 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.417567 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":96,"threads_started":1}
I20260924 17:47:37.417719 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.413166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39303 (local address 127.0.0.1:42630)
0924 17:47:37.413768 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.413778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.413841 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.414135 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.414136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.414138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:37.414174 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.414175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.415449 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.415450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.416545 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.416548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.417491 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.417492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.417547 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.417550 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.417609 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.417646 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":59,"threads_started":1}
I20260924 17:47:37.417796 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.413324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.1.8.130:39303)
0924 17:47:37.413517 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.413521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.413912 (+ 391us) server_negotiation.cc:407] Connection header received
0924 17:47:37.413961 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.413964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.413982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.414053 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:37.414249 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.415346 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.416634 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.417347 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.417407 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.417457 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.417648 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.417704 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260924 17:47:37.418418 18220 tablet_service.cc:1467] Processing CreateTablet for tablet 7744f8d8f1e44f559e01666db4c04029 (DEFAULT_TABLE table=test-table [id=3f54787ac3ca443cbf3da082c5847d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:37.417441 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.412930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45267 (local address 127.0.0.1:38612)
0924 17:47:37.413279 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.413290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.413356 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.413691 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.413692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.413694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:37.413917 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.413918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.415285 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.415286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.416281 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.416282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.417192 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.417193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.417257 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.417261 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.417313 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.417358 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":85,"threads_started":1}
I20260924 17:47:37.418558 18354 tablet_service.cc:1467] Processing CreateTablet for tablet 7744f8d8f1e44f559e01666db4c04029 (DEFAULT_TABLE table=test-table [id=3f54787ac3ca443cbf3da082c5847d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:37.418812 18220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7744f8d8f1e44f559e01666db4c04029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:37.418913 18354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7744f8d8f1e44f559e01666db4c04029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:37.419405 18488 tablet_service.cc:1467] Processing CreateTablet for tablet 7744f8d8f1e44f559e01666db4c04029 (DEFAULT_TABLE table=test-table [id=3f54787ac3ca443cbf3da082c5847d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:37.419703 18488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7744f8d8f1e44f559e01666db4c04029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:37.421371 18571 tablet_bootstrap.cc:492] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: Bootstrap starting.
I20260924 17:47:37.422046 18571 tablet_bootstrap.cc:654] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:37.422360 18572 tablet_bootstrap.cc:492] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Bootstrap starting.
I20260924 17:47:37.422712 18571 log.cc:824] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:37.423066 18573 tablet_bootstrap.cc:492] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Bootstrap starting.
I20260924 17:47:37.423167 18572 tablet_bootstrap.cc:654] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:37.423746 18572 log.cc:824] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:37.423784 18573 tablet_bootstrap.cc:654] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:37.423983 18571 tablet_bootstrap.cc:492] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: No bootstrap required, opened a new log
I20260924 17:47:37.424062 18571 ts_tablet_manager.cc:1397] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:47:37.424477 18573 log.cc:824] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:37.424567 18572 tablet_bootstrap.cc:492] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: No bootstrap required, opened a new log
I20260924 17:47:37.424669 18572 ts_tablet_manager.cc:1397] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 17:47:37.425163 18573 tablet_bootstrap.cc:492] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: No bootstrap required, opened a new log
I20260924 17:47:37.425249 18573 ts_tablet_manager.cc:1397] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:47:37.426148 18571 raft_consensus.cc:348] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.426355 18571 raft_consensus.cc:374] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:37.426424 18571 raft_consensus.cc:729] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18f2fb5d1724f85a19e6e57120f8655, State: Initialized, Role: FOLLOWER
I20260924 17:47:37.426555 18571 consensus_queue.cc:260] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.426867 18571 ts_tablet_manager.cc:1428] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:37.426915 18286 heartbeater.cc:503] Master 127.1.8.190:35501 was elected leader, sending a full tablet report...
I20260924 17:47:37.426798 18573 raft_consensus.cc:348] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.426968 18573 raft_consensus.cc:374] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:37.426995 18573 raft_consensus.cc:729] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24878f326eb94822abe246148e605fa4, State: Initialized, Role: FOLLOWER
I20260924 17:47:37.427102 18573 consensus_queue.cc:260] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.427414 18573 ts_tablet_manager.cc:1428] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:47:37.427505 18554 heartbeater.cc:503] Master 127.1.8.190:35501 was elected leader, sending a full tablet report...
I20260924 17:47:37.426714 18572 raft_consensus.cc:348] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.427723 18572 raft_consensus.cc:374] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:37.427819 18572 raft_consensus.cc:729] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5dfe96647b4f6387a69a0fe80713a6, State: Initialized, Role: FOLLOWER
I20260924 17:47:37.427945 18572 consensus_queue.cc:260] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.428249 18420 heartbeater.cc:503] Master 127.1.8.190:35501 was elected leader, sending a full tablet report...
I20260924 17:47:37.428272 18572 ts_tablet_manager.cc:1428] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
W20260924 17:47:37.480511 18421 tablet.cc:2369] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:47:37.562073 18287 tablet.cc:2369] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:47:37.638517 18555 tablet.cc:2369] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:47:37.675985 18579 raft_consensus.cc:482] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:37.676155 18579 raft_consensus.cc:504] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.676560 18579 leader_election.cc:288] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24878f326eb94822abe246148e605fa4 (127.1.8.131:45267), b18f2fb5d1724f85a19e6e57120f8655 (127.1.8.129:33623)
I20260924 17:47:37.679747 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.676898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:58253 (local address 127.1.8.129:33623)
0924 17:47:37.676947 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.676949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.676963 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:47:37.676997 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.676998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.677011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.677091 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:37.677378 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.677875 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.679008 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.679359 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.679417 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.679426 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.679621 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.679670 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 17:47:37.679747 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.676792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33623 (local address 127.1.8.130:58253)
0924 17:47:37.676864 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.676878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.676925 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.677107 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.677108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.677110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:37.677258 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.677260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.677987 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.677988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.678878 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.678881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.679448 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.679449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.679535 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.679538 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.679591 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.679653 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":112}
I20260924 17:47:37.679766 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.676792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:45267 (local address 127.1.8.130:50737)
0924 17:47:37.677000 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.677011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:37.677079 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:37.677229 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:37.677230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:37.677231 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:37.677290 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.677292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.677963 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.677964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.678869 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:37.678871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.679501 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.679502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:37.679559 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.679563 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:37.679648 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 17:47:37.679686 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":23,"thread_start_us":118,"threads_started":1}
I20260924 17:47:37.679816 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.676913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:50737 (local address 127.1.8.131:45267)
0924 17:47:37.677014 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.677016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.677089 (+ 73us) server_negotiation.cc:407] Connection header received
0924 17:47:37.677136 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.677137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.677150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.677213 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:37.677407 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.677866 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.678974 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.679382 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.679480 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.679490 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.679675 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.679739 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 17:47:37.680140 18240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7744f8d8f1e44f559e01666db4c04029" candidate_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18f2fb5d1724f85a19e6e57120f8655" is_pre_election: true
I20260924 17:47:37.680331 18240 raft_consensus.cc:2454] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5dfe96647b4f6387a69a0fe80713a6 in term 0.
I20260924 17:47:37.680276 18508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7744f8d8f1e44f559e01666db4c04029" candidate_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24878f326eb94822abe246148e605fa4" is_pre_election: true
I20260924 17:47:37.680469 18508 raft_consensus.cc:2454] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b5dfe96647b4f6387a69a0fe80713a6 in term 0.
I20260924 17:47:37.680615 18307 leader_election.cc:302] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5dfe96647b4f6387a69a0fe80713a6, b18f2fb5d1724f85a19e6e57120f8655; no voters:
I20260924 17:47:37.680775 18579 raft_consensus.cc:2781] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:47:37.680835 18579 raft_consensus.cc:482] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:47:37.680868 18579 raft_consensus.cc:3037] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:37.681641 18579 raft_consensus.cc:504] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.681802 18579 leader_election.cc:288] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [CANDIDATE]: Term 1 election: Requested vote from peers 24878f326eb94822abe246148e605fa4 (127.1.8.131:45267), b18f2fb5d1724f85a19e6e57120f8655 (127.1.8.129:33623)
I20260924 17:47:37.682019 18508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7744f8d8f1e44f559e01666db4c04029" candidate_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24878f326eb94822abe246148e605fa4"
I20260924 17:47:37.682026 18240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7744f8d8f1e44f559e01666db4c04029" candidate_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18f2fb5d1724f85a19e6e57120f8655"
I20260924 17:47:37.682096 18508 raft_consensus.cc:3037] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:37.682148 18240 raft_consensus.cc:3037] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:37.682955 18240 raft_consensus.cc:2454] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5dfe96647b4f6387a69a0fe80713a6 in term 1.
I20260924 17:47:37.682945 18508 raft_consensus.cc:2454] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b5dfe96647b4f6387a69a0fe80713a6 in term 1.
I20260924 17:47:37.683187 18307 leader_election.cc:302] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b5dfe96647b4f6387a69a0fe80713a6, b18f2fb5d1724f85a19e6e57120f8655; no voters:
I20260924 17:47:37.683303 18579 raft_consensus.cc:2781] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:37.683526 18579 raft_consensus.cc:686] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 1 LEADER]: Becoming Leader. State: Replica: 8b5dfe96647b4f6387a69a0fe80713a6, State: Running, Role: LEADER
I20260924 17:47:37.683706 18579 consensus_queue.cc:237] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } }
I20260924 17:47:37.684648 18094 catalog_manager.cc:5495] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b5dfe96647b4f6387a69a0fe80713a6 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b5dfe96647b4f6387a69a0fe80713a6" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39303 } health_report { overall_health: HEALTHY } } }
I20260924 17:47:37.703729 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.701401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.1.8.130:39303)
0924 17:47:37.701473 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.701475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.701483 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:47:37.701517 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.701518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.701537 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.701597 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:47:37.701701 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.702184 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.703040 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.703243 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.703302 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.703422 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:47:37.703527 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:47:37.703587 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.703635 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.703657 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 17:47:37.705608 18508 raft_consensus.cc:1261] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 1 FOLLOWER]: Refusing update from remote peer 8b5dfe96647b4f6387a69a0fe80713a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:37.705617 18240 raft_consensus.cc:1261] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 1 FOLLOWER]: Refusing update from remote peer 8b5dfe96647b4f6387a69a0fe80713a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:37.705986 18579 consensus_queue.cc:1035] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24878f326eb94822abe246148e605fa4" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 45267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:37.706144 18581 consensus_queue.cc:1035] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18f2fb5d1724f85a19e6e57120f8655" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 33623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:37.709695 18586 mvcc.cc:205] Tried to move back new op lower bound from 7332954348359204864 to 7332954348273057792. Current Snapshot: MvccSnapshot[applied={T|T < 7332954348359204864}]
I20260924 17:47:37.711063 18094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:37.711170 18094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:37.711984 18094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c57e93fecc4040f9ac3454c1b9f70d1d: Sending DeleteTablet for 3 replicas of tablet 7744f8d8f1e44f559e01666db4c04029
I20260924 17:47:37.712316 18488 tablet_service.cc:1514] Processing DeleteTablet for tablet 7744f8d8f1e44f559e01666db4c04029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:37 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:38612
I20260924 17:47:37.712373 18354 tablet_service.cc:1514] Processing DeleteTablet for tablet 7744f8d8f1e44f559e01666db4c04029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:37 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:42630
I20260924 17:47:37.712549 18220 tablet_service.cc:1514] Processing DeleteTablet for tablet 7744f8d8f1e44f559e01666db4c04029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:37 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:43846
I20260924 17:47:37.712603 18594 tablet_replica.cc:330] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: stopping tablet replica
I20260924 17:47:37.712702 18594 raft_consensus.cc:2229] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:47:37.712764 18593 tablet_replica.cc:330] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: stopping tablet replica
I20260924 17:47:37.712801 18594 raft_consensus.cc:2258] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:37.712875 18593 raft_consensus.cc:2229] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:37.712980 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18157
I20260924 17:47:37.713016 18595 tablet_replica.cc:330] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655: stopping tablet replica
I20260924 17:47:37.713109 18595 raft_consensus.cc:2229] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:37.713214 18595 raft_consensus.cc:2258] T 7744f8d8f1e44f559e01666db4c04029 P b18f2fb5d1724f85a19e6e57120f8655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:37.713349 18593 raft_consensus.cc:2258] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:37.713896 18593 ts_tablet_manager.cc:1905] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:37.713933 18594 ts_tablet_manager.cc:1905] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:37.715950 18593 ts_tablet_manager.cc:1918] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:37.715950 18594 ts_tablet_manager.cc:1918] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:37.716022 18594 log.cc:1196] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/7744f8d8f1e44f559e01666db4c04029
I20260924 17:47:37.716022 18593 log.cc:1196] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/7744f8d8f1e44f559e01666db4c04029
I20260924 17:47:37.716288 18593 ts_tablet_manager.cc:1939] T 7744f8d8f1e44f559e01666db4c04029 P 24878f326eb94822abe246148e605fa4: Deleting consensus metadata
I20260924 17:47:37.716288 18594 ts_tablet_manager.cc:1939] T 7744f8d8f1e44f559e01666db4c04029 P 8b5dfe96647b4f6387a69a0fe80713a6: Deleting consensus metadata
I20260924 17:47:37.716686 18080 catalog_manager.cc:4841] TS 24878f326eb94822abe246148e605fa4 (127.1.8.131:45267): tablet 7744f8d8f1e44f559e01666db4c04029 (table test-table [id=3f54787ac3ca443cbf3da082c5847d01]) successfully deleted
I20260924 17:47:37.717208 18080 catalog_manager.cc:4841] TS 8b5dfe96647b4f6387a69a0fe80713a6 (127.1.8.130:39303): tablet 7744f8d8f1e44f559e01666db4c04029 (table test-table [id=3f54787ac3ca443cbf3da082c5847d01]) successfully deleted
W20260924 17:47:37.721540 18079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:33623 (error 108)
I20260924 17:47:37.722149 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.721989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:33623 (local address 127.0.0.1:43854)
0924 17:47:37.722045 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:37.722083 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:47:37.722215 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18291
W20260924 17:47:37.722292 18079 catalog_manager.cc:4584] TS b18f2fb5d1724f85a19e6e57120f8655 (127.1.8.129:33623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7744f8d8f1e44f559e01666db4c04029: Network error: Client connection negotiation failed: client connection to 127.1.8.129:33623: connect: Connection refused (error 111)
I20260924 17:47:37.728070 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18425
I20260924 17:47:37.732954 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18063
2026-09-24T17:47:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2171 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T17:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:37Z Disabled control of system clock
I20260924 17:47:37.760907 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37561
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:37561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:37.842991 18600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:37.843212 18600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:37.843233 18600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:37.844345 18600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:37.844381 18600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:37.844394 18600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:37.844408 18600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:37.844419 18600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:37.845860 18600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:37561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:37.846098 18600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:37.846349 18600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:37.851456 18606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.851573 18600 server_base.cc:1034] running on GCE node
W20260924 17:47:37.851492 18605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:37.851613 18608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:37.851943 18600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:37.852257 18600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:37.853428 18600 hybrid_clock.cc:648] HybridClock initialized: now 1790272057853429 us; error 47 us; skew 500 ppm
I20260924 17:47:37.854761 18600 webserver.cc:460] Webserver started at http://127.1.8.190:42889/ using document root <none> and password file <none>
I20260924 17:47:37.855010 18600 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:37.855078 18600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:37.855235 18600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:37.856401 18600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "6c549133cbe14241bcc61781c2c605ca"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "aa6e0c74ef7df3d4be8ac73450a2c9b8"
server_key_iv: "ae198bdfd86b79658e8f4ee1473aead0"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.856767 18600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "6c549133cbe14241bcc61781c2c605ca"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "aa6e0c74ef7df3d4be8ac73450a2c9b8"
server_key_iv: "ae198bdfd86b79658e8f4ee1473aead0"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.858091 18600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:37.858897 18614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.859059 18600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:37.859167 18600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "6c549133cbe14241bcc61781c2c605ca"
format_stamp: "Formatted at 2026-09-24 17:47:37 on dist-test-slave-r7hf"
server_key: "aa6e0c74ef7df3d4be8ac73450a2c9b8"
server_key_iv: "ae198bdfd86b79658e8f4ee1473aead0"
server_key_version: "encryptionkey@0"
I20260924 17:47:37.859243 18600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:37.872953 18600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:37.873303 18600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:37.873459 18600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:37.877403 18600 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:37561
I20260924 17:47:37.877466 18666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:37561 every 8 connection(s)
I20260924 17:47:37.877836 18600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:37.878968 18667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:37.881281 18667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: Bootstrap starting.
I20260924 17:47:37.881913 18667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:37.882220 18667 log.cc:824] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:37.882961 18667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: No bootstrap required, opened a new log
I20260924 17:47:37.884706 18667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c549133cbe14241bcc61781c2c605ca" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37561 } }
I20260924 17:47:37.884856 18667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:37.884879 18667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c549133cbe14241bcc61781c2c605ca, State: Initialized, Role: FOLLOWER
I20260924 17:47:37.885058 18667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c549133cbe14241bcc61781c2c605ca" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37561 } }
I20260924 17:47:37.885123 18667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:37.885146 18667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:37.885210 18667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:37.885521 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18600
I20260924 17:47:37.885600 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:37.885805 1058 external_mini_cluster.cc:1428] Setting key 8044265ec557d9fe94a0ed1e7a88e392
I20260924 17:47:37.885857 18667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c549133cbe14241bcc61781c2c605ca" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37561 } }
I20260924 17:47:37.885967 18667 leader_election.cc:302] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c549133cbe14241bcc61781c2c605ca; no voters:
I20260924 17:47:37.886188 18667 leader_election.cc:288] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:37.886497 18670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:37.886526 18667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:37.887250 18670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [term 1 LEADER]: Becoming Leader. State: Replica: 6c549133cbe14241bcc61781c2c605ca, State: Running, Role: LEADER
I20260924 17:47:37.887372 18670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c549133cbe14241bcc61781c2c605ca" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37561 } }
I20260924 17:47:37.888031 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c549133cbe14241bcc61781c2c605ca. Latest consensus state: current_term: 1 leader_uuid: "6c549133cbe14241bcc61781c2c605ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c549133cbe14241bcc61781c2c605ca" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37561 } } }
I20260924 17:47:37.887948 18672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c549133cbe14241bcc61781c2c605ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c549133cbe14241bcc61781c2c605ca" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37561 } } }
I20260924 17:47:37.888180 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:37.888199 18672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca [sys.catalog]: This master's current role is: LEADER
W20260924 17:47:37.888553 18686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:47:37.888609 18686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:47:37.888676 18688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:37.888844 18688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:37.890380 18688 catalog_manager.cc:1347] Generated new cluster ID: 69605e0f9a31459bb9da1607010bfe5c
I20260924 17:47:37.890462 18688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:37.892552 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:37.886332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.1.8.190:37561)
0924 17:47:37.886551 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:37.886560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:37.886595 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:47:37.886716 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:37.886724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:37.886794 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:37.886937 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:37.887158 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.889334 (+ 2176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.890493 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:37.890941 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:37.891087 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:37.891851 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:37.891855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:37.891855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:37.892036 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:37.892068 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:37.892080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:37.892161 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:37.892239 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 17:47:37.892336 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260924 17:47:37.909279 18688 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:37.909865 18688 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:37.916522 18688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: Generated new TSK 0
I20260924 17:47:37.916749 18688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:37.922333 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37561
--builtin_ntp_servers=127.1.8.148:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:38.007028 18691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:38.007257 18691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:38.007328 18691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:38.008592 18691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:38.008642 18691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:38.008670 18691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:38.010102 18691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:38.010422 18691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:38.010757 18691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:38.016930 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:38.017128 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:38.017030 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:38.017485 18691 server_base.cc:1034] running on GCE node
I20260924 17:47:38.017694 18691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:38.017956 18691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:38.019130 18691 hybrid_clock.cc:648] HybridClock initialized: now 1790272058019116 us; error 30 us; skew 500 ppm
I20260924 17:47:38.020511 18691 webserver.cc:460] Webserver started at http://127.1.8.129:45877/ using document root <none> and password file <none>
I20260924 17:47:38.020774 18691 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:38.020860 18691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:38.021016 18691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:38.022120 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "54aba1a5d07342a89ef217b0c9813556"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "b4081091db6521a4e33429438a8780e1"
server_key_iv: "aebfd49b399983e3ab2ec4a7f9295dbd"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.022547 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "54aba1a5d07342a89ef217b0c9813556"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "b4081091db6521a4e33429438a8780e1"
server_key_iv: "aebfd49b399983e3ab2ec4a7f9295dbd"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.023900 18691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:38.024701 18705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.024888 18691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:38.024988 18691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "54aba1a5d07342a89ef217b0c9813556"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "b4081091db6521a4e33429438a8780e1"
server_key_iv: "aebfd49b399983e3ab2ec4a7f9295dbd"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.025058 18691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:38.060781 18691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:38.061138 18691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:38.061301 18691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:38.061556 18691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:38.061925 18691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:38.061995 18691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.062034 18691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:38.062084 18691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.068113 18691 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:41715
I20260924 17:47:38.068169 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:41715 every 8 connection(s)
I20260924 17:47:38.068630 18691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:47:38.074303 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.069468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:45433 (local address 127.1.8.190:37561)
0924 17:47:38.069543 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.069545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.069791 (+ 246us) server_negotiation.cc:407] Connection header received
0924 17:47:38.070368 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.070369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.070386 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.070472 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:38.071184 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.071704 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.072716 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.072941 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.073020 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.074016 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:38.074018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:38.074018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:38.074024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:38.074038 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:38.074043 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:38.074092 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.074186 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.074221 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:47:38.074560 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.069446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37561 (local address 127.1.8.129:45433)
0924 17:47:38.069664 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.069699 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.070235 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.070553 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.070557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.070565 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:38.071077 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.071081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.071813 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.071814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.072617 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.072619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.073040 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.073042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.073230 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.073866 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:38.073866 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:38.073896 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:38.073921 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:38.074091 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:38.074096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:38.074099 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.074189 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.074292 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260924 17:47:38.075107 18819 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37561
I20260924 17:47:38.075192 18819 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:38.075398 18819 heartbeater.cc:511] Master 127.1.8.190:37561 requested a full tablet report, sending...
I20260924 17:47:38.075928 18631 ts_manager.cc:194] Registered new tserver with Master: 54aba1a5d07342a89ef217b0c9813556 (127.1.8.129:41715)
I20260924 17:47:38.076557 18631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.129:45433
I20260924 17:47:38.078115 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18691
I20260924 17:47:38.078239 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:47:38.078478 1058 external_mini_cluster.cc:1428] Setting key 9e223abbf14f0b8ec91e0369a0adaacb
I20260924 17:47:38.079429 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37561
--builtin_ntp_servers=127.1.8.148:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:38.161522 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:38.161798 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:38.161868 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:38.163187 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:38.163262 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:38.163309 18822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:38.164721 18822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:38.165016 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:38.165333 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:38.170608 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:38.170614 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:38.170608 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:38.170976 18822 server_base.cc:1034] running on GCE node
I20260924 17:47:38.171243 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:38.171475 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:38.172659 18822 hybrid_clock.cc:648] HybridClock initialized: now 1790272058172645 us; error 29 us; skew 500 ppm
I20260924 17:47:38.174074 18822 webserver.cc:460] Webserver started at http://127.1.8.130:42963/ using document root <none> and password file <none>
I20260924 17:47:38.174343 18822 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:38.174445 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:38.174604 18822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:38.175737 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "57176ab597164d2d922a891b2e59b027"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "a0790a62321a802727314d9a49094600"
server_key_iv: "81bae633551f426ed28b9f4850b3a185"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.176126 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "57176ab597164d2d922a891b2e59b027"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "a0790a62321a802727314d9a49094600"
server_key_iv: "81bae633551f426ed28b9f4850b3a185"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.177522 18822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:38.178300 18836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.178515 18822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:38.178612 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "57176ab597164d2d922a891b2e59b027"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "a0790a62321a802727314d9a49094600"
server_key_iv: "81bae633551f426ed28b9f4850b3a185"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.178696 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:38.213402 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:38.213706 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:38.213819 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:38.214003 18822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:38.214293 18822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:38.214325 18822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.214349 18822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:38.214360 18822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.220688 18822 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:40639
I20260924 17:47:38.220723 18949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:40639 every 8 connection(s)
I20260924 17:47:38.221118 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:38.224812 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18822
I20260924 17:47:38.224906 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:47:38.225117 1058 external_mini_cluster.cc:1428] Setting key 8a5320481830aa0d0d1b67b063236c2a
I20260924 17:47:38.226112 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37561
--builtin_ntp_servers=127.1.8.148:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:47:38.226866 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.221976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:33869 (local address 127.1.8.190:37561)
0924 17:47:38.222045 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.222047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.222261 (+ 214us) server_negotiation.cc:407] Connection header received
0924 17:47:38.222864 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.222864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.222874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.222937 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:38.223617 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.224112 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.225289 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.225494 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.225575 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.226557 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:38.226559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:38.226560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:38.226565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:38.226587 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:38.226591 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:38.226648 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.226755 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.226787 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:47:38.227110 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.221935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37561 (local address 127.1.8.130:33869)
0924 17:47:38.222129 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.222171 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.222762 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.223016 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.223020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.223028 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:38.223539 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.223543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.224219 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.224220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.225176 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.225180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.225584 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.225585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.225726 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.226362 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:38.226363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:38.226428 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:38.226463 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:38.226663 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:38.226668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:38.226672 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.226758 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.226887 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 17:47:38.227715 18950 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37561
I20260924 17:47:38.227845 18950 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:38.228060 18950 heartbeater.cc:511] Master 127.1.8.190:37561 requested a full tablet report, sending...
I20260924 17:47:38.228504 18631 ts_manager.cc:194] Registered new tserver with Master: 57176ab597164d2d922a891b2e59b027 (127.1.8.130:40639)
I20260924 17:47:38.229039 18631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.130:33869
W20260924 17:47:38.312629 18953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:38.312853 18953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:38.312876 18953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:38.313970 18953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:38.314003 18953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:38.314028 18953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:38.315348 18953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:38.315572 18953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:38.315833 18953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:47:38.321223 18953 server_base.cc:1034] running on GCE node
W20260924 17:47:38.321203 18959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:38.321225 18958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:38.321197 18961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:38.321704 18953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:38.321982 18953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:38.323150 18953 hybrid_clock.cc:648] HybridClock initialized: now 1790272058323135 us; error 46 us; skew 500 ppm
I20260924 17:47:38.324422 18953 webserver.cc:460] Webserver started at http://127.1.8.131:42375/ using document root <none> and password file <none>
I20260924 17:47:38.324676 18953 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:38.324747 18953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:38.324911 18953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:38.326032 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "a1eb06d54b454252997b6dc6ce1d5560"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "fcf8de962a4fd1d01e84f1e95efef681"
server_key_iv: "1be3a595ba5b136a45ed692cc4f9d2ac"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.326460 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "a1eb06d54b454252997b6dc6ce1d5560"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "fcf8de962a4fd1d01e84f1e95efef681"
server_key_iv: "1be3a595ba5b136a45ed692cc4f9d2ac"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.327937 18953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:38.328755 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.328938 18953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:38.329056 18953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "a1eb06d54b454252997b6dc6ce1d5560"
format_stamp: "Formatted at 2026-09-24 17:47:38 on dist-test-slave-r7hf"
server_key: "fcf8de962a4fd1d01e84f1e95efef681"
server_key_iv: "1be3a595ba5b136a45ed692cc4f9d2ac"
server_key_version: "encryptionkey@0"
I20260924 17:47:38.329136 18953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:38.351635 18953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:38.352010 18953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:38.352169 18953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:38.352440 18953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:38.352900 18953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:38.352959 18953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.353004 18953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:38.353029 18953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:38.358994 18953 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:33473
I20260924 17:47:38.359061 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:33473 every 8 connection(s)
I20260924 17:47:38.359424 18953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:47:38.361858 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 18953
I20260924 17:47:38.361963 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:47:38.362149 1058 external_mini_cluster.cc:1428] Setting key d6d2f4bc0065fbfa34aedbc374d4dcab
I20260924 17:47:38.365414 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.360171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:39661 (local address 127.1.8.190:37561)
0924 17:47:38.360218 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.360220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.360485 (+ 265us) server_negotiation.cc:407] Connection header received
0924 17:47:38.361075 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.361076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.361091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.361142 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:38.361991 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.362829 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.364023 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.364257 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.364335 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.365170 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:38.365171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:38.365172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:38.365176 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:38.365189 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:38.365193 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:38.365257 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.365328 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.365354 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:47:38.365672 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.360157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37561 (local address 127.1.8.131:39661)
0924 17:47:38.360348 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.360385 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.360936 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.361215 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.361219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.361230 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:38.361882 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.361888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.363043 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.363044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.363917 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.363919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.364347 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.364348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.364462 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.364996 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:38.364997 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:38.365028 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:38.365053 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:38.365260 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:38.365262 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:38.365265 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.365314 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.365396 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260924 17:47:38.366214 19081 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37561
I20260924 17:47:38.366300 19081 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:38.366531 19081 heartbeater.cc:511] Master 127.1.8.190:37561 requested a full tablet report, sending...
I20260924 17:47:38.366986 18631 ts_manager.cc:194] Registered new tserver with Master: a1eb06d54b454252997b6dc6ce1d5560 (127.1.8.131:33473)
I20260924 17:47:38.367339 18631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.131:39661
I20260924 17:47:38.374328 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:47:38.378789 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.375779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.1.8.190:37561)
0924 17:47:38.375882 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.375884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.375927 (+ 43us) server_negotiation.cc:407] Connection header received
0924 17:47:38.375971 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.375971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.375983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.376031 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:38.376276 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.376743 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.377586 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.377764 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.377815 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.378513 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:38.378515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:38.378515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:38.378520 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:38.378532 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:38.378536 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:38.378578 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.378646 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.378668 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 17:47:38.380276 18631 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:47:38.381956 18631 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:47:38.389103 19089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.384610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:33473 (local address 127.0.0.1:49018)
0924 17:47:38.384790 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.384798 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.384840 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.385089 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.385092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.385095 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:38.385166 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.385168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.386632 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.386633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.388062 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.388065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.388785 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.388786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.388868 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.388884 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.388964 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.389018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":44,"threads_started":1}
I20260924 17:47:38.389112 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.384728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.1.8.131:33473)
0924 17:47:38.384926 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.384931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.384940 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:38.384979 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.384981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.385000 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.385062 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:38.385252 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.386562 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.388151 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.388706 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.388778 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.388872 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.388979 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.389017 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260924 17:47:38.390228 19015 tablet_service.cc:1467] Processing CreateTablet for tablet c6dbf7e8587d4397b50574faa0394c92 (DEFAULT_TABLE table=test-table [id=d27a4fa6d9df44dabf52c29390d54d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:38.390666 19015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6dbf7e8587d4397b50574faa0394c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:38.390941 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.384387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:40639 (local address 127.0.0.1:57534)
0924 17:47:38.384720 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.384764 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.384841 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.385555 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.385556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.385558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:38.386224 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.386227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.388710 (+ 2483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.388711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.389903 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.389906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.390713 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.390714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.390773 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.390776 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.390821 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.390864 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":85,"thread_start_us":74,"threads_started":1}
I20260924 17:47:38.390960 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.384580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.1.8.130:40639)
0924 17:47:38.385304 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.385319 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.385358 (+ 39us) server_negotiation.cc:407] Connection header received
0924 17:47:38.385402 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.385404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.385448 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.385557 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:38.386317 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.387490 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.389993 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.390611 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.390709 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.390802 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.390839 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.390885 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":111,"threads_started":1}
I20260924 17:47:38.391402 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.384390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:41715 (local address 127.0.0.1:40676)
0924 17:47:38.384733 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.384765 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.384840 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.385221 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.385222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.385224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:38.386217 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.386220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.388714 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.388715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.389957 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.389960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.390938 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.390939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.391010 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.391015 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.391274 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.391321 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":31,"thread_start_us":73,"threads_started":1}
I20260924 17:47:38.391664 18884 tablet_service.cc:1467] Processing CreateTablet for tablet c6dbf7e8587d4397b50574faa0394c92 (DEFAULT_TABLE table=test-table [id=d27a4fa6d9df44dabf52c29390d54d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:38.391775 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.384505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.1.8.129:41715)
0924 17:47:38.384982 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.384986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.384994 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:47:38.385025 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.385026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.385038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.385153 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:38.387596 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.388626 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.390058 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.390835 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.391507 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.391613 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.391641 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.391681 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":92,"threads_started":1}
I20260924 17:47:38.392023 18884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6dbf7e8587d4397b50574faa0394c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:38.392483 18753 tablet_service.cc:1467] Processing CreateTablet for tablet c6dbf7e8587d4397b50574faa0394c92 (DEFAULT_TABLE table=test-table [id=d27a4fa6d9df44dabf52c29390d54d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:38.392805 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6dbf7e8587d4397b50574faa0394c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:38.394601 19095 tablet_bootstrap.cc:492] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Bootstrap starting.
I20260924 17:47:38.395226 19096 tablet_bootstrap.cc:492] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Bootstrap starting.
I20260924 17:47:38.395387 19095 tablet_bootstrap.cc:654] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:38.395730 19095 log.cc:824] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:38.395912 19096 tablet_bootstrap.cc:654] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:38.396270 19096 log.cc:824] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:38.396529 19095 tablet_bootstrap.cc:492] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: No bootstrap required, opened a new log
I20260924 17:47:38.396601 19095 ts_tablet_manager.cc:1397] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:47:38.396600 19097 tablet_bootstrap.cc:492] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556: Bootstrap starting.
I20260924 17:47:38.396941 19096 tablet_bootstrap.cc:492] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: No bootstrap required, opened a new log
I20260924 17:47:38.397028 19096 ts_tablet_manager.cc:1397] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:47:38.397243 19097 tablet_bootstrap.cc:654] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:38.397603 19097 log.cc:824] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:38.398779 19097 tablet_bootstrap.cc:492] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556: No bootstrap required, opened a new log
I20260924 17:47:38.398869 19097 ts_tablet_manager.cc:1397] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:47:38.399192 19095 raft_consensus.cc:348] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.399398 19095 raft_consensus.cc:374] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:38.399312 19096 raft_consensus.cc:348] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.399444 19095 raft_consensus.cc:729] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1eb06d54b454252997b6dc6ce1d5560, State: Initialized, Role: FOLLOWER
I20260924 17:47:38.399499 19096 raft_consensus.cc:374] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:38.399566 19096 raft_consensus.cc:729] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57176ab597164d2d922a891b2e59b027, State: Initialized, Role: FOLLOWER
I20260924 17:47:38.399576 19095 consensus_queue.cc:260] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.399792 19096 consensus_queue.cc:260] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.399967 19081 heartbeater.cc:503] Master 127.1.8.190:37561 was elected leader, sending a full tablet report...
I20260924 17:47:38.399978 19095 ts_tablet_manager.cc:1428] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 17:47:38.400059 19096 ts_tablet_manager.cc:1428] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:38.400166 18950 heartbeater.cc:503] Master 127.1.8.190:37561 was elected leader, sending a full tablet report...
I20260924 17:47:38.400410 19103 raft_consensus.cc:482] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:38.400532 19103 raft_consensus.cc:504] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.401942 19103 leader_election.cc:288] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54aba1a5d07342a89ef217b0c9813556 (127.1.8.129:41715), a1eb06d54b454252997b6dc6ce1d5560 (127.1.8.131:33473)
I20260924 17:47:38.401839 19097 raft_consensus.cc:348] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.402001 19097 raft_consensus.cc:374] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:38.402062 19097 raft_consensus.cc:729] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54aba1a5d07342a89ef217b0c9813556, State: Initialized, Role: FOLLOWER
I20260924 17:47:38.402241 19097 consensus_queue.cc:260] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.402657 19097 ts_tablet_manager.cc:1428] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 17:47:38.402827 18819 heartbeater.cc:503] Master 127.1.8.190:37561 was elected leader, sending a full tablet report...
I20260924 17:47:38.404717 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.401092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:38489 (local address 127.1.8.129:41715)
0924 17:47:38.401140 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.401143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.401252 (+ 109us) server_negotiation.cc:407] Connection header received
0924 17:47:38.401276 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.401277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.401297 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.401365 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:38.401519 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.402630 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.403935 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.404358 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.404453 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.404463 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.404617 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.404653 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:47:38.404718 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.400980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:41715 (local address 127.1.8.130:38489)
0924 17:47:38.401137 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.401149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.401198 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.401381 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.401383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.401384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:38.401440 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.401441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.402715 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.402716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.403863 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.403865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.404456 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.404457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.404525 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.404528 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.404609 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.404651 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260924 17:47:38.404793 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.401732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:48645 (local address 127.1.8.131:33473)
0924 17:47:38.401772 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.401774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.401787 (+ 13us) server_negotiation.cc:407] Connection header received
0924 17:47:38.401811 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.401811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.401841 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.401907 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:38.402054 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.402729 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.404070 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.404419 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.404505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.404514 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.404687 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.404726 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:47:38.404831 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.401534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:33473 (local address 127.1.8.130:48645)
0924 17:47:38.401690 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.401701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.401748 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.401912 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:38.401913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:38.401914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:38.401974 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.401975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.402829 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.402830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.403993 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:38.403995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.404508 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.404509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:38.404587 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.404591 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:38.404680 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:47:38.404717 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260924 17:47:38.405100 19035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dbf7e8587d4397b50574faa0394c92" candidate_uuid: "57176ab597164d2d922a891b2e59b027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1eb06d54b454252997b6dc6ce1d5560" is_pre_election: true
I20260924 17:47:38.405237 19035 raft_consensus.cc:2454] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57176ab597164d2d922a891b2e59b027 in term 0.
I20260924 17:47:38.405468 18839 leader_election.cc:302] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57176ab597164d2d922a891b2e59b027, a1eb06d54b454252997b6dc6ce1d5560; no voters:
I20260924 17:47:38.405450 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dbf7e8587d4397b50574faa0394c92" candidate_uuid: "57176ab597164d2d922a891b2e59b027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54aba1a5d07342a89ef217b0c9813556" is_pre_election: true
I20260924 17:47:38.405550 18773 raft_consensus.cc:2454] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57176ab597164d2d922a891b2e59b027 in term 0.
I20260924 17:47:38.405613 19101 raft_consensus.cc:2781] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:47:38.405696 19101 raft_consensus.cc:482] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:47:38.405741 19101 raft_consensus.cc:3037] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:38.406486 19101 raft_consensus.cc:504] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.406654 19101 leader_election.cc:288] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [CANDIDATE]: Term 1 election: Requested vote from peers 54aba1a5d07342a89ef217b0c9813556 (127.1.8.129:41715), a1eb06d54b454252997b6dc6ce1d5560 (127.1.8.131:33473)
I20260924 17:47:38.406803 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dbf7e8587d4397b50574faa0394c92" candidate_uuid: "57176ab597164d2d922a891b2e59b027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54aba1a5d07342a89ef217b0c9813556"
I20260924 17:47:38.406813 19035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6dbf7e8587d4397b50574faa0394c92" candidate_uuid: "57176ab597164d2d922a891b2e59b027" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1eb06d54b454252997b6dc6ce1d5560"
I20260924 17:47:38.406893 18773 raft_consensus.cc:3037] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:38.406893 19035 raft_consensus.cc:3037] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:38.407689 18773 raft_consensus.cc:2454] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57176ab597164d2d922a891b2e59b027 in term 1.
I20260924 17:47:38.407720 19035 raft_consensus.cc:2454] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57176ab597164d2d922a891b2e59b027 in term 1.
I20260924 17:47:38.407873 18839 leader_election.cc:302] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57176ab597164d2d922a891b2e59b027, a1eb06d54b454252997b6dc6ce1d5560; no voters:
I20260924 17:47:38.408004 19101 raft_consensus.cc:2781] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:38.408072 19101 raft_consensus.cc:686] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 1 LEADER]: Becoming Leader. State: Replica: 57176ab597164d2d922a891b2e59b027, State: Running, Role: LEADER
I20260924 17:47:38.408157 19101 consensus_queue.cc:237] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } }
I20260924 17:47:38.409026 18631 catalog_manager.cc:5495] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 reported cstate change: term changed from 0 to 1, leader changed from <none> to 57176ab597164d2d922a891b2e59b027 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "57176ab597164d2d922a891b2e59b027" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57176ab597164d2d922a891b2e59b027" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 40639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 } health_report { overall_health: UNKNOWN } } }
I20260924 17:47:38.418530 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:38.416084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.1.8.130:40639)
0924 17:47:38.416217 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:38.416219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:38.416228 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:38.416263 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:38.416263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:38.416282 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:38.416357 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:47:38.416472 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.416953 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.417800 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:38.418001 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:38.418063 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:38.418186 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:47:38.418285 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:47:38.418350 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:38.418382 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 17:47:38.418461 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260924 17:47:38.420418 18773 raft_consensus.cc:1261] T c6dbf7e8587d4397b50574faa0394c92 P 54aba1a5d07342a89ef217b0c9813556 [term 1 FOLLOWER]: Refusing update from remote peer 57176ab597164d2d922a891b2e59b027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:38.420456 19035 raft_consensus.cc:1261] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 1 FOLLOWER]: Refusing update from remote peer 57176ab597164d2d922a891b2e59b027: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:38.420781 19101 consensus_queue.cc:1035] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54aba1a5d07342a89ef217b0c9813556" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 41715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:38.420925 19103 consensus_queue.cc:1035] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1eb06d54b454252997b6dc6ce1d5560" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 33473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:38.423491 19110 mvcc.cc:205] Tried to move back new op lower bound from 7332954351287017472 to 7332954351240564736. Current Snapshot: MvccSnapshot[applied={T|T < 7332954351287017472}]
I20260924 17:47:38.423755 19109 mvcc.cc:205] Tried to move back new op lower bound from 7332954351287017472 to 7332954351240564736. Current Snapshot: MvccSnapshot[applied={T|T < 7332954351287017472}]
I20260924 17:47:38.425506 18631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:38.425601 18631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:38.426272 18631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c549133cbe14241bcc61781c2c605ca: Sending DeleteTablet for 3 replicas of tablet c6dbf7e8587d4397b50574faa0394c92
I20260924 17:47:38.426695 18753 tablet_service.cc:1514] Processing DeleteTablet for tablet c6dbf7e8587d4397b50574faa0394c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:38 UTC) from {username='slave'} at 127.0.0.1:40676
I20260924 17:47:38.426748 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet c6dbf7e8587d4397b50574faa0394c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:38 UTC) from {username='slave'} at 127.0.0.1:57534
I20260924 17:47:38.426904 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18691
I20260924 17:47:38.426956 19015 tablet_service.cc:1514] Processing DeleteTablet for tablet c6dbf7e8587d4397b50574faa0394c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:38 UTC) from {username='slave'} at 127.0.0.1:49018
I20260924 17:47:38.427570 19118 tablet_replica.cc:330] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: stopping tablet replica
I20260924 17:47:38.427666 19118 raft_consensus.cc:2229] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:47:38.427789 19118 raft_consensus.cc:2258] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:38.427874 19119 tablet_replica.cc:330] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: stopping tablet replica
I20260924 17:47:38.427968 19119 raft_consensus.cc:2229] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:38.428067 19119 raft_consensus.cc:2258] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:38.428344 19118 ts_tablet_manager.cc:1905] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:38.429905 19118 ts_tablet_manager.cc:1918] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:38.429957 19118 log.cc:1196] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/c6dbf7e8587d4397b50574faa0394c92
I20260924 17:47:38.430132 19118 ts_tablet_manager.cc:1939] T c6dbf7e8587d4397b50574faa0394c92 P 57176ab597164d2d922a891b2e59b027: Deleting consensus metadata
I20260924 17:47:38.430675 18618 catalog_manager.cc:4841] TS 57176ab597164d2d922a891b2e59b027 (127.1.8.130:40639): tablet c6dbf7e8587d4397b50574faa0394c92 (table test-table [id=d27a4fa6d9df44dabf52c29390d54d65]) successfully deleted
W20260924 17:47:38.434122 18617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:41715 (error 108)
I20260924 17:47:38.434779 19119 ts_tablet_manager.cc:1905] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 17:47:38.436177 19092 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:47:38.434571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:41715 (local address 127.0.0.1:40692)
0924 17:47:38.434635 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:38.434648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:38.434698 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:38.436119 (+ 1421us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:41715: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:47:38.436333 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18822
W20260924 17:47:38.436332 18617 catalog_manager.cc:4584] TS 54aba1a5d07342a89ef217b0c9813556 (127.1.8.129:41715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6dbf7e8587d4397b50574faa0394c92: Network error: Client connection negotiation failed: client connection to 127.1.8.129:41715: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 17:47:38.438541 19119 ts_tablet_manager.cc:1918] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:38.438612 19119 log.cc:1196] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/c6dbf7e8587d4397b50574faa0394c92
I20260924 17:47:38.438886 19119 ts_tablet_manager.cc:1939] T c6dbf7e8587d4397b50574faa0394c92 P a1eb06d54b454252997b6dc6ce1d5560: Deleting consensus metadata
I20260924 17:47:38.439344 18617 catalog_manager.cc:4841] TS a1eb06d54b454252997b6dc6ce1d5560 (127.1.8.131:33473): tablet c6dbf7e8587d4397b50574faa0394c92 (table test-table [id=d27a4fa6d9df44dabf52c29390d54d65]) successfully deleted
I20260924 17:47:38.441732 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18953
I20260924 17:47:38.447127 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 18600
2026-09-24T17:47:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (712 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:38 dist-test-slave-r7hf krb5kdc[19123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:38 dist-test-slave-r7hf krb5kdc[19123](info): set up 2 sockets
Sep 24 17:47:38 dist-test-slave-r7hf krb5kdc[19123](info): commencing operation
krb5kdc: starting...
W20260924 17:47:39.510157 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:39 dist-test-slave-r7hf krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:47:39.638088 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:46085
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:46085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:39.723069 19136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:39.723331 19136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:39.723369 19136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:39.724583 19136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:39.724659 19136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:39.724695 19136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:39.724726 19136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:39.724750 19136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:39.726276 19136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:46085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:39.726591 19136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:39.726832 19136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:39.731961 19144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:39.731945 19142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:39.731949 19141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:39.731982 19136 server_base.cc:1034] running on GCE node
I20260924 17:47:39.732483 19136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:39.732833 19136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:39.734004 19136 hybrid_clock.cc:648] HybridClock initialized: now 1790272059733983 us; error 36 us; skew 500 ppm
Sep 24 17:47:39 dist-test-slave-r7hf krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:39.736575 19136 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:47:39.737035 19136 webserver.cc:460] Webserver started at http://127.1.8.190:45719/ using document root <none> and password file <none>
I20260924 17:47:39.737278 19136 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:39.737349 19136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:39.737496 19136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:39.738655 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "1b11362084e94f3394312e2734d31260"
format_stamp: "Formatted at 2026-09-24 17:47:39 on dist-test-slave-r7hf"
server_key: "8fe4a7787077d557be868f3ab096a7c5"
server_key_iv: "d5623f2cea8568f68525defd23d74318"
server_key_version: "encryptionkey@0"
I20260924 17:47:39.739020 19136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "1b11362084e94f3394312e2734d31260"
format_stamp: "Formatted at 2026-09-24 17:47:39 on dist-test-slave-r7hf"
server_key: "8fe4a7787077d557be868f3ab096a7c5"
server_key_iv: "d5623f2cea8568f68525defd23d74318"
server_key_version: "encryptionkey@0"
I20260924 17:47:39.740427 19136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:39.741273 19151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:39.741521 19136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:39.741633 19136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "1b11362084e94f3394312e2734d31260"
format_stamp: "Formatted at 2026-09-24 17:47:39 on dist-test-slave-r7hf"
server_key: "8fe4a7787077d557be868f3ab096a7c5"
server_key_iv: "d5623f2cea8568f68525defd23d74318"
server_key_version: "encryptionkey@0"
I20260924 17:47:39.741712 19136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:39.752279 19136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:39.755107 19136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:39.755302 19136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:39.759236 19136 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:46085
I20260924 17:47:39.759289 19203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:46085 every 8 connection(s)
I20260924 17:47:39.759690 19136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:39.760565 19204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:39.763008 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: Bootstrap starting.
I20260924 17:47:39.763235 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19136
I20260924 17:47:39.763348 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:39.763567 1058 external_mini_cluster.cc:1428] Setting key a5ce8d525a5dff7d94aca5109abc8def
I20260924 17:47:39.763808 19204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:39.764147 19204 log.cc:824] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:39.764922 19204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: No bootstrap required, opened a new log
I20260924 17:47:39.767318 19204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b11362084e94f3394312e2734d31260" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46085 } }
I20260924 17:47:39.767504 19204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:39.767534 19204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b11362084e94f3394312e2734d31260, State: Initialized, Role: FOLLOWER
I20260924 17:47:39.767690 19204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b11362084e94f3394312e2734d31260" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46085 } }
I20260924 17:47:39.767767 19204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:39.767791 19204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:39.767863 19204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:39.768503 19204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b11362084e94f3394312e2734d31260" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46085 } }
I20260924 17:47:39.768615 19204 leader_election.cc:302] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b11362084e94f3394312e2734d31260; no voters:
I20260924 17:47:39.768778 19204 leader_election.cc:288] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:39.768846 19209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:39.769035 19209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [term 1 LEADER]: Becoming Leader. State: Replica: 1b11362084e94f3394312e2734d31260, State: Running, Role: LEADER
I20260924 17:47:39.769040 19204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:39.769898 19209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b11362084e94f3394312e2734d31260" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46085 } }
Sep 24 17:47:39 dist-test-slave-r7hf krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
W20260924 17:47:39.771023 19223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 17:47:39.771225 19211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b11362084e94f3394312e2734d31260. Latest consensus state: current_term: 1 leader_uuid: "1b11362084e94f3394312e2734d31260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b11362084e94f3394312e2734d31260" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46085 } } }
W20260924 17:47:39.771920 19223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:47:39.771947 19211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:39.772212 19226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:39.772447 19226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:39.771356 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b11362084e94f3394312e2734d31260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b11362084e94f3394312e2734d31260" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46085 } } }
I20260924 17:47:39.772578 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:39.774116 19226 catalog_manager.cc:1347] Generated new cluster ID: ac192cd781264e229ec80ac2b5de4604
I20260924 17:47:39.774166 19226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:39.776341 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:39.764156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.1.8.190:46085)
0924 17:47:39.764352 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:39.764362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:39.764395 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:47:39.764989 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:39.764998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:39.765073 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:39.765231 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:39.765851 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:39.767356 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:39.768305 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:39.768954 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:39.769111 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:39.772241 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:39.772246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:39.772246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:39.772260 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:39.774426 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:39.774831 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:39.774834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:39.774835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:39.774931 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:39.775095 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:39.775096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:39.775096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:39.775374 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:39.775513 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:39.776004 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 17:47:39.776139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260924 17:47:39.805588 19226 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:39.806378 19226 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:39.810660 19226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: Generated new TSK 0
I20260924 17:47:39.810877 19226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:47:39.841837 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46085
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:39.925467 19230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:39.925689 19230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:39.925714 19230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:39.927047 19230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:39.927094 19230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:39.927124 19230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:39.928763 19230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:39.929028 19230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:39.929276 19230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:39.934623 19235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:39.934628 19238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:39.934638 19236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:39.934974 19230 server_base.cc:1034] running on GCE node
I20260924 17:47:39.935225 19230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:39.935570 19230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:39.936781 19230 hybrid_clock.cc:648] HybridClock initialized: now 1790272059936770 us; error 37 us; skew 500 ppm
Sep 24 17:47:39 dist-test-slave-r7hf krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:39.939827 19230 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:47:39.940366 19230 webserver.cc:460] Webserver started at http://127.1.8.129:44255/ using document root <none> and password file <none>
I20260924 17:47:39.940655 19230 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:39.940739 19230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:39.940904 19230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:39.942170 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "b31958d47ffb48628e8890ebb4d17b4e"
format_stamp: "Formatted at 2026-09-24 17:47:39 on dist-test-slave-r7hf"
server_key: "e709b9cf85632ba5673b24ecbd088712"
server_key_iv: "67ce6435a09f25425d7d32e6c3cd7c0c"
server_key_version: "encryptionkey@0"
I20260924 17:47:39.942648 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "b31958d47ffb48628e8890ebb4d17b4e"
format_stamp: "Formatted at 2026-09-24 17:47:39 on dist-test-slave-r7hf"
server_key: "e709b9cf85632ba5673b24ecbd088712"
server_key_iv: "67ce6435a09f25425d7d32e6c3cd7c0c"
server_key_version: "encryptionkey@0"
I20260924 17:47:39.944234 19230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:39.945271 19245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:39.945513 19230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:39.945667 19230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "b31958d47ffb48628e8890ebb4d17b4e"
format_stamp: "Formatted at 2026-09-24 17:47:39 on dist-test-slave-r7hf"
server_key: "e709b9cf85632ba5673b24ecbd088712"
server_key_iv: "67ce6435a09f25425d7d32e6c3cd7c0c"
server_key_version: "encryptionkey@0"
I20260924 17:47:39.945783 19230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:39.960381 19230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:39.962841 19230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:39.963027 19230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:39.963315 19230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:39.963665 19230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:39.963722 19230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:39.963773 19230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:39.963797 19230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:39.970077 19230 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:39885
I20260924 17:47:39.970113 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:39885 every 8 connection(s)
I20260924 17:47:39.970561 19230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:47:39.977201 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19230
I20260924 17:47:39.977324 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:47:39.977600 1058 external_mini_cluster.cc:1428] Setting key cd2393e5af49018f4d110ec69722ad38
Sep 24 17:47:39 dist-test-slave-r7hf krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272059, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:39.982118 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:39.971682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:49809 (local address 127.1.8.190:46085)
0924 17:47:39.971825 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:39.971828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:39.972036 (+ 208us) server_negotiation.cc:407] Connection header received
0924 17:47:39.972672 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:39.972674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:39.972687 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:39.972757 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:39.973845 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:39.974503 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:39.975673 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:39.975909 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:39.976009 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:39.978927 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:39.978928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:39.978929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:39.978936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:39.980489 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:39.981024 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:39.981024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:39.981025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:39.981075 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:39.981378 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:39.981380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:39.981380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:39.981533 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:39.981638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:39.981879 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 17:47:39.981982 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 17:47:39.982146 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:39.971602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:46085 (local address 127.1.8.129:49809)
0924 17:47:39.971860 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:39.971923 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:39.972523 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:39.972843 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:39.972850 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:39.973493 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:39.973728 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:39.973733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:39.974644 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:39.974645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:39.975558 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:39.975561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:39.976021 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:39.976022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:39.976187 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:39.976816 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:39.976827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:39.978826 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:39.980580 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:39.980589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:39.980592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:39.980943 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:39.981141 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:39.981142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:39.981142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:39.981304 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:39.981600 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:39.981605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:39.981737 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 17:47:39.981837 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 17:47:39.981975 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":113,"threads_started":1}
I20260924 17:47:39.982710 19359 heartbeater.cc:347] Connected to a master server at 127.1.8.190:46085
I20260924 17:47:39.982817 19359 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:39.983039 19359 heartbeater.cc:511] Master 127.1.8.190:46085 requested a full tablet report, sending...
I20260924 17:47:39.983635 19168 ts_manager.cc:194] Registered new tserver with Master: b31958d47ffb48628e8890ebb4d17b4e (127.1.8.129:39885)
I20260924 17:47:39.984283 19168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:49809
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:47:40.010236 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46085
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:40.098877 19364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:40.099153 19364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:40.099197 19364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:40.100387 19364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:40.100464 19364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:40.100509 19364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:40.102012 19364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:40.102303 19364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:40.102581 19364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:40.107767 19369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.107760 19370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.107995 19364 server_base.cc:1034] running on GCE node
W20260924 17:47:40.107790 19372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.108408 19364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:40.108654 19364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:40.109849 19364 hybrid_clock.cc:648] HybridClock initialized: now 1790272060109830 us; error 34 us; skew 500 ppm
Sep 24 17:47:40 dist-test-slave-r7hf krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:40.112537 19364 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:47:40.113063 19364 webserver.cc:460] Webserver started at http://127.1.8.130:40091/ using document root <none> and password file <none>
I20260924 17:47:40.113301 19364 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:40.113371 19364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:40.113533 19364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:40.114728 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "721129546dd343e1b90d6dc1c59e8419"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "5955edbeccfc37c56e6422c155166b3c"
server_key_iv: "3def880bb1eeb91eba53b8daf74d18e0"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.115118 19364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "721129546dd343e1b90d6dc1c59e8419"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "5955edbeccfc37c56e6422c155166b3c"
server_key_iv: "3def880bb1eeb91eba53b8daf74d18e0"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.116504 19364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:40.117307 19379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.117515 19364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:40.117624 19364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "721129546dd343e1b90d6dc1c59e8419"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "5955edbeccfc37c56e6422c155166b3c"
server_key_iv: "3def880bb1eeb91eba53b8daf74d18e0"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.117713 19364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:40.132253 19364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:40.134202 19364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:40.134377 19364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:40.134718 19364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:40.135061 19364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:40.135119 19364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.135165 19364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:40.135190 19364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.141402 19364 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:39361
I20260924 17:47:40.141449 19492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:39361 every 8 connection(s)
I20260924 17:47:40.141796 19364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:40.145639 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19364
I20260924 17:47:40.145735 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:47:40.145967 1058 external_mini_cluster.cc:1428] Setting key 737fc794e6d61def444e08eb7f3c4116
Sep 24 17:47:40 dist-test-slave-r7hf krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:40.153152 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.142608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:50223 (local address 127.1.8.190:46085)
0924 17:47:40.142657 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.142659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.142932 (+ 273us) server_negotiation.cc:407] Connection header received
0924 17:47:40.143541 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.143543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.143557 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.143615 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.144758 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.145318 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.146682 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.146928 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.147018 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.149721 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.149722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.149723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:40.149732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.151607 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:40.152057 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:40.152059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:40.152060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:40.152129 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:40.152396 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:40.152397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:40.152398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:40.152569 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.152650 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.152910 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.153013 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:47:40.153173 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.142610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:46085 (local address 127.1.8.130:50223)
0924 17:47:40.142800 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.142840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.143408 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.143700 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.143704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.144428 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:40.144642 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.144648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.145431 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.145433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.146591 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.146594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.147029 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.147030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.147168 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.147752 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:40.147762 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:40.149598 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:40.151695 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:40.151703 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:40.151705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:40.151966 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:40.152205 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:40.152205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:40.152206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:40.152328 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:40.152640 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:40.152644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:40.152768 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.152857 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.152992 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260924 17:47:40.153795 19493 heartbeater.cc:347] Connected to a master server at 127.1.8.190:46085
I20260924 17:47:40.153915 19493 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:40.154129 19493 heartbeater.cc:511] Master 127.1.8.190:46085 requested a full tablet report, sending...
I20260924 17:47:40.154564 19168 ts_manager.cc:194] Registered new tserver with Master: 721129546dd343e1b90d6dc1c59e8419 (127.1.8.130:39361)
I20260924 17:47:40.154958 19168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:50223
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:47:40.175704 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46085
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:40.257755 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:40.257974 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:40.258000 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:40.259236 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:40.259284 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:40.259347 19498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:40.260815 19498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:46085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:40.261076 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:40.261310 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:40.266484 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.266520 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.266642 19498 server_base.cc:1034] running on GCE node
W20260924 17:47:40.266516 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.266898 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:40.267179 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:40.268339 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790272060268318 us; error 39 us; skew 500 ppm
Sep 24 17:47:40 dist-test-slave-r7hf krb5kdc[19123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:40.271229 19498 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:47:40.271720 19498 webserver.cc:460] Webserver started at http://127.1.8.131:34013/ using document root <none> and password file <none>
I20260924 17:47:40.271932 19498 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:40.271975 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:40.272066 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:40.273118 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "31cf9f324e4a45e79201ed39e6944758"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "3581c1f837f3513909c2f15834c0b26a"
server_key_iv: "87f8b0519359bb7b96462191721d0e6f"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.273423 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "31cf9f324e4a45e79201ed39e6944758"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "3581c1f837f3513909c2f15834c0b26a"
server_key_iv: "87f8b0519359bb7b96462191721d0e6f"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.274734 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:40.275539 19513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.275758 19498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:40.275832 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "31cf9f324e4a45e79201ed39e6944758"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "3581c1f837f3513909c2f15834c0b26a"
server_key_iv: "87f8b0519359bb7b96462191721d0e6f"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.275889 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:40.295666 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:40.297590 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:40.297708 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:40.297909 19498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:40.298234 19498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:40.298271 19498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.298293 19498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:40.298341 19498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.304442 19498 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:34909
I20260924 17:47:40.304479 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:34909 every 8 connection(s)
I20260924 17:47:40.304844 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:47:40 dist-test-slave-r7hf krb5kdc[19123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272060, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:40.311167 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19498
I20260924 17:47:40.311272 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:47:40.311455 1058 external_mini_cluster.cc:1428] Setting key 1fabebd21dd97b1323e8db721eea9840
I20260924 17:47:40.314826 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.305629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:42057 (local address 127.1.8.190:46085)
0924 17:47:40.305702 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.305709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.305935 (+ 226us) server_negotiation.cc:407] Connection header received
0924 17:47:40.306517 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.306518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.306529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.306598 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.307574 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.308078 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.309074 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.309280 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.309336 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.311718 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.311719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.311720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:40.311728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.313279 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:40.313724 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:40.313726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:40.313726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:40.313789 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:40.314062 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:40.314063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:40.314064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:40.314186 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.314280 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.314638 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.314723 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:47:40.315057 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.305622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:46085 (local address 127.1.8.131:42057)
0924 17:47:40.305799 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.305844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.306350 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.306661 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.306665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.307283 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:40.307482 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.307487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.308165 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.308166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.308989 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.308991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.309358 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.309359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.309460 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.309948 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:40.309957 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:40.311608 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:40.313389 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:40.313397 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:40.313400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:40.313644 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:40.313866 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:40.313867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:40.313867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:40.313989 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:40.314273 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:40.314280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:40.314494 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.314617 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.314807 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260924 17:47:40.315582 19627 heartbeater.cc:347] Connected to a master server at 127.1.8.190:46085
I20260924 17:47:40.315703 19627 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:40.315900 19627 heartbeater.cc:511] Master 127.1.8.190:46085 requested a full tablet report, sending...
I20260924 17:47:40.316296 19168 ts_manager.cc:194] Registered new tserver with Master: 31cf9f324e4a45e79201ed39e6944758 (127.1.8.131:34909)
I20260924 17:47:40.316696 19168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:42057
I20260924 17:47:40.323268 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:47:40.331264 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.324658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.1.8.190:46085)
0924 17:47:40.324741 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.324744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.324837 (+ 93us) server_negotiation.cc:407] Connection header received
0924 17:47:40.324906 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.324907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.324917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.324967 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.325560 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.326097 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.326933 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.327138 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.327214 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.328610 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.328611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.328612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:40.328621 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.329860 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:40.330303 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:40.330304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:40.330305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:40.330369 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:40.330645 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:40.330647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:40.330647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:40.330815 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.330937 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.331083 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.331155 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 17:47:40.332993 19168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:47:40.334674 19168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:47:40.341751 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.337415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.1.8.129:39885)
0924 17:47:40.337587 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.337591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.337625 (+ 34us) server_negotiation.cc:407] Connection header received
0924 17:47:40.337716 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.337718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.337733 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.338015 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.338086 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.339259 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.340551 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.341321 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.341433 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.341499 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.341636 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.341673 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":75,"threads_started":1}
I20260924 17:47:40.341776 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.337104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39885 (local address 127.0.0.1:50152)
0924 17:47:40.337562 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.337574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.337641 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.337836 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.337837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.337839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.337934 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.337937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.339367 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.339368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.340468 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.340470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.341434 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.341435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.341526 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.341541 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.341614 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.341672 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":94,"threads_started":1}
I20260924 17:47:40.342195 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.337099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34909 (local address 127.0.0.1:59142)
0924 17:47:40.337279 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.337289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.337341 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.337968 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.337969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.337971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.338024 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.338026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.339073 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.339074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.340438 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.340440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.341899 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.341900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.341998 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.342002 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.342058 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.342101 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":37,"thread_start_us":74,"threads_started":1}
I20260924 17:47:40.342505 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.337590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.1.8.131:34909)
0924 17:47:40.337763 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.337801 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.337812 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:47:40.337849 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.337851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.337867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.337972 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.338079 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.338952 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.340557 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.341321 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.341433 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.341499 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.342305 (+ 806us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.342355 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260924 17:47:40.342638 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.337099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:39361 (local address 127.0.0.1:38128)
0924 17:47:40.337224 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.337268 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.337341 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.337776 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.337779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.337781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.337858 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.337860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.340558 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.340559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.341312 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.341314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.342385 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.342386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.342472 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.342476 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.342535 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.342572 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":90,"threads_started":1}
I20260924 17:47:40.342710 19293 tablet_service.cc:1467] Processing CreateTablet for tablet 80dc18074fce4f87b4259b93b9e8e912 (DEFAULT_TABLE table=test-table [id=615a4e76fcc4423c814403d45ae92f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:40.343039 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.337374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.1.8.130:39361)
0924 17:47:40.337571 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.337577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.337588 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:47:40.337631 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.337633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.337649 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.337763 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.338135 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.339259 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.341399 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.342157 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.342268 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.342334 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.342914 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.342958 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260924 17:47:40.343112 19293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80dc18074fce4f87b4259b93b9e8e912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:40.343415 19561 tablet_service.cc:1467] Processing CreateTablet for tablet 80dc18074fce4f87b4259b93b9e8e912 (DEFAULT_TABLE table=test-table [id=615a4e76fcc4423c814403d45ae92f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:40.343766 19561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80dc18074fce4f87b4259b93b9e8e912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:40.343884 19427 tablet_service.cc:1467] Processing CreateTablet for tablet 80dc18074fce4f87b4259b93b9e8e912 (DEFAULT_TABLE table=test-table [id=615a4e76fcc4423c814403d45ae92f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:40.344228 19427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80dc18074fce4f87b4259b93b9e8e912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:40.346859 19644 tablet_bootstrap.cc:492] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e: Bootstrap starting.
I20260924 17:47:40.347078 19645 tablet_bootstrap.cc:492] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Bootstrap starting.
I20260924 17:47:40.347563 19646 tablet_bootstrap.cc:492] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Bootstrap starting.
I20260924 17:47:40.347676 19644 tablet_bootstrap.cc:654] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:40.347860 19645 tablet_bootstrap.cc:654] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:40.348304 19644 log.cc:824] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:40.348335 19646 tablet_bootstrap.cc:654] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:40.348348 19645 log.cc:824] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:40.349057 19646 log.cc:824] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:40.349287 19644 tablet_bootstrap.cc:492] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e: No bootstrap required, opened a new log
I20260924 17:47:40.349341 19645 tablet_bootstrap.cc:492] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: No bootstrap required, opened a new log
I20260924 17:47:40.349412 19645 ts_tablet_manager.cc:1397] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:47:40.349447 19644 ts_tablet_manager.cc:1397] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:47:40.351090 19646 tablet_bootstrap.cc:492] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: No bootstrap required, opened a new log
I20260924 17:47:40.351176 19646 ts_tablet_manager.cc:1397] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 17:47:40.351305 19644 raft_consensus.cc:348] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.351297 19645 raft_consensus.cc:348] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.351445 19644 raft_consensus.cc:374] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:40.351466 19644 raft_consensus.cc:729] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31958d47ffb48628e8890ebb4d17b4e, State: Initialized, Role: FOLLOWER
I20260924 17:47:40.351457 19645 raft_consensus.cc:374] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:40.351491 19645 raft_consensus.cc:729] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cf9f324e4a45e79201ed39e6944758, State: Initialized, Role: FOLLOWER
I20260924 17:47:40.351583 19644 consensus_queue.cc:260] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.351622 19645 consensus_queue.cc:260] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.351879 19645 ts_tablet_manager.cc:1428] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 17:47:40.351863 19359 heartbeater.cc:503] Master 127.1.8.190:46085 was elected leader, sending a full tablet report...
I20260924 17:47:40.352025 19627 heartbeater.cc:503] Master 127.1.8.190:46085 was elected leader, sending a full tablet report...
I20260924 17:47:40.351859 19644 ts_tablet_manager.cc:1428] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:47:40.353222 19646 raft_consensus.cc:348] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.353404 19646 raft_consensus.cc:374] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:40.353432 19646 raft_consensus.cc:729] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721129546dd343e1b90d6dc1c59e8419, State: Initialized, Role: FOLLOWER
I20260924 17:47:40.353677 19646 consensus_queue.cc:260] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.354017 19646 ts_tablet_manager.cc:1428] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:40.354040 19493 heartbeater.cc:503] Master 127.1.8.190:46085 was elected leader, sending a full tablet report...
I20260924 17:47:40.357882 19652 raft_consensus.cc:482] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:40.357975 19652 raft_consensus.cc:504] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.358270 19652 leader_election.cc:288] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31cf9f324e4a45e79201ed39e6944758 (127.1.8.131:34909), b31958d47ffb48628e8890ebb4d17b4e (127.1.8.129:39885)
I20260924 17:47:40.361500 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.358439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39885 (local address 127.1.8.130:42755)
0924 17:47:40.358662 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.358669 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.358805 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.359010 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.359011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.359013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.359076 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.359078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.359676 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.359676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.360759 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.360762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.361298 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.361299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.361353 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.361356 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.361402 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.361432 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":10,"thread_start_us":98,"threads_started":1}
I20260924 17:47:40.361544 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.358550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:42755 (local address 127.1.8.129:39885)
0924 17:47:40.358619 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.358621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.358732 (+ 111us) server_negotiation.cc:407] Connection header received
0924 17:47:40.358901 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.358902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.358915 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.358965 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.359138 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.359605 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.360846 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.361227 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.361285 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.361294 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.361437 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.361478 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 17:47:40.361830 19313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dc18074fce4f87b4259b93b9e8e912" candidate_uuid: "721129546dd343e1b90d6dc1c59e8419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31958d47ffb48628e8890ebb4d17b4e" is_pre_election: true
I20260924 17:47:40.362001 19313 raft_consensus.cc:2454] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721129546dd343e1b90d6dc1c59e8419 in term 0.
I20260924 17:47:40.362249 19382 leader_election.cc:302] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 721129546dd343e1b90d6dc1c59e8419, b31958d47ffb48628e8890ebb4d17b4e; no voters:
I20260924 17:47:40.362303 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.358437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:34909 (local address 127.1.8.130:47097)
0924 17:47:40.358491 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.358504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.358547 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.358704 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.358706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.358707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.358788 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.358789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.359530 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.359531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.360761 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.360762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.362036 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.362037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.362124 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.362128 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.362186 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.362218 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:47:40.362571 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.358508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:47097 (local address 127.1.8.131:34909)
0924 17:47:40.358557 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.358560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.358571 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:47:40.358621 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.358622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.358631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.358699 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.358864 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.359448 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.361325 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.361922 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.361993 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.362004 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.362470 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.362510 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:47:40.362689 19652 raft_consensus.cc:2781] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:47:40.362776 19652 raft_consensus.cc:482] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:47:40.362811 19652 raft_consensus.cc:3037] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:40.362864 19581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dc18074fce4f87b4259b93b9e8e912" candidate_uuid: "721129546dd343e1b90d6dc1c59e8419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cf9f324e4a45e79201ed39e6944758" is_pre_election: true
I20260924 17:47:40.363019 19581 raft_consensus.cc:2454] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 721129546dd343e1b90d6dc1c59e8419 in term 0.
I20260924 17:47:40.363641 19652 raft_consensus.cc:504] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.363798 19652 leader_election.cc:288] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [CANDIDATE]: Term 1 election: Requested vote from peers 31cf9f324e4a45e79201ed39e6944758 (127.1.8.131:34909), b31958d47ffb48628e8890ebb4d17b4e (127.1.8.129:39885)
I20260924 17:47:40.363932 19581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dc18074fce4f87b4259b93b9e8e912" candidate_uuid: "721129546dd343e1b90d6dc1c59e8419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cf9f324e4a45e79201ed39e6944758"
I20260924 17:47:40.363969 19313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80dc18074fce4f87b4259b93b9e8e912" candidate_uuid: "721129546dd343e1b90d6dc1c59e8419" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31958d47ffb48628e8890ebb4d17b4e"
I20260924 17:47:40.364029 19581 raft_consensus.cc:3037] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:40.364030 19313 raft_consensus.cc:3037] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:40.364711 19581 raft_consensus.cc:2454] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721129546dd343e1b90d6dc1c59e8419 in term 1.
I20260924 17:47:40.364812 19313 raft_consensus.cc:2454] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 721129546dd343e1b90d6dc1c59e8419 in term 1.
I20260924 17:47:40.364904 19381 leader_election.cc:302] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31cf9f324e4a45e79201ed39e6944758, 721129546dd343e1b90d6dc1c59e8419; no voters:
I20260924 17:47:40.365005 19652 raft_consensus.cc:2781] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:40.365191 19652 raft_consensus.cc:686] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 1 LEADER]: Becoming Leader. State: Replica: 721129546dd343e1b90d6dc1c59e8419, State: Running, Role: LEADER
I20260924 17:47:40.365291 19652 consensus_queue.cc:237] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } }
I20260924 17:47:40.366134 19168 catalog_manager.cc:5495] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 reported cstate change: term changed from 0 to 1, leader changed from <none> to 721129546dd343e1b90d6dc1c59e8419 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "721129546dd343e1b90d6dc1c59e8419" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "721129546dd343e1b90d6dc1c59e8419" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 39361 } health_report { overall_health: HEALTHY } } }
I20260924 17:47:40.377755 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.375519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.1.8.130:39361)
0924 17:47:40.375588 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.375590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.375598 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:47:40.375630 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.375631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.375649 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.375712 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:47:40.375810 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.376262 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.376972 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.377151 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.377201 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.377327 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:47:40.377413 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:47:40.377486 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.377659 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.377667 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 17:47:40.379863 19313 raft_consensus.cc:1261] T 80dc18074fce4f87b4259b93b9e8e912 P b31958d47ffb48628e8890ebb4d17b4e [term 1 FOLLOWER]: Refusing update from remote peer 721129546dd343e1b90d6dc1c59e8419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:40.379878 19581 raft_consensus.cc:1261] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 1 FOLLOWER]: Refusing update from remote peer 721129546dd343e1b90d6dc1c59e8419: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:40.380263 19654 consensus_queue.cc:1035] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cf9f324e4a45e79201ed39e6944758" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:40.380420 19652 consensus_queue.cc:1035] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31958d47ffb48628e8890ebb4d17b4e" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:40.383013 19657 mvcc.cc:205] Tried to move back new op lower bound from 7332954359313010688 to 7332954359256895488. Current Snapshot: MvccSnapshot[applied={T|T < 7332954359313010688}]
I20260924 17:47:40.383221 19658 mvcc.cc:205] Tried to move back new op lower bound from 7332954359313010688 to 7332954359256895488. Current Snapshot: MvccSnapshot[applied={T|T < 7332954359313010688}]
I20260924 17:47:40.383924 19659 mvcc.cc:205] Tried to move back new op lower bound from 7332954359313010688 to 7332954359256895488. Current Snapshot: MvccSnapshot[applied={T|T < 7332954359313010688}]
I20260924 17:47:40.385649 19166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:40.385756 19166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:40.386576 19166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b11362084e94f3394312e2734d31260: Sending DeleteTablet for 3 replicas of tablet 80dc18074fce4f87b4259b93b9e8e912
I20260924 17:47:40.386909 19561 tablet_service.cc:1514] Processing DeleteTablet for tablet 80dc18074fce4f87b4259b93b9e8e912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:40 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:59142
I20260924 17:47:40.386938 19293 tablet_service.cc:1514] Processing DeleteTablet for tablet 80dc18074fce4f87b4259b93b9e8e912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:40 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:50152
I20260924 17:47:40.387145 19427 tablet_service.cc:1514] Processing DeleteTablet for tablet 80dc18074fce4f87b4259b93b9e8e912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:40 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:38128
I20260924 17:47:40.387215 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19230
I20260924 17:47:40.388499 19669 tablet_replica.cc:330] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: stopping tablet replica
I20260924 17:47:40.388615 19669 raft_consensus.cc:2229] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:47:40.388768 19669 raft_consensus.cc:2258] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:40.389259 19669 ts_tablet_manager.cc:1905] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:40.391412 19669 ts_tablet_manager.cc:1918] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:40.391491 19669 log.cc:1196] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/80dc18074fce4f87b4259b93b9e8e912
I20260924 17:47:40.391777 19669 ts_tablet_manager.cc:1939] T 80dc18074fce4f87b4259b93b9e8e912 P 721129546dd343e1b90d6dc1c59e8419: Deleting consensus metadata
I20260924 17:47:40.392228 19152 catalog_manager.cc:4841] TS 721129546dd343e1b90d6dc1c59e8419 (127.1.8.130:39361): tablet 80dc18074fce4f87b4259b93b9e8e912 (table test-table [id=615a4e76fcc4423c814403d45ae92f52]) successfully deleted
W20260924 17:47:40.393451 19154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:39885 (error 108)
I20260924 17:47:40.393697 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19364
I20260924 17:47:40.393937 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.393821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:39885 (local address 127.0.0.1:50158)
0924 17:47:40.393870 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.393899 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:39885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 17:47:40.394040 19154 catalog_manager.cc:4584] TS b31958d47ffb48628e8890ebb4d17b4e (127.1.8.129:39885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80dc18074fce4f87b4259b93b9e8e912: Network error: Client connection negotiation failed: client connection to 127.1.8.129:39885: connect: Connection refused (error 111)
I20260924 17:47:40.394358 19667 tablet_replica.cc:330] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: stopping tablet replica
I20260924 17:47:40.394520 19667 raft_consensus.cc:2229] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:40.394651 19667 raft_consensus.cc:2258] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:40.396386 19667 ts_tablet_manager.cc:1905] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:40.398008 19667 ts_tablet_manager.cc:1918] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:40.398074 19667 log.cc:1196] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/80dc18074fce4f87b4259b93b9e8e912
I20260924 17:47:40.398342 19667 ts_tablet_manager.cc:1939] T 80dc18074fce4f87b4259b93b9e8e912 P 31cf9f324e4a45e79201ed39e6944758: Deleting consensus metadata
I20260924 17:47:40.398844 19153 catalog_manager.cc:4841] TS 31cf9f324e4a45e79201ed39e6944758 (127.1.8.131:34909): tablet 80dc18074fce4f87b4259b93b9e8e912 (table test-table [id=615a4e76fcc4423c814403d45ae92f52]) successfully deleted
I20260924 17:47:40.399865 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19498
I20260924 17:47:40.405412 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19136
2026-09-24T17:47:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T17:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:40Z Disabled control of system clock
I20260924 17:47:40.433305 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44421
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:40.514699 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:40.514945 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:40.514966 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:40.516088 19674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:40.516126 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:40.516139 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:40.516151 19674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:40.516163 19674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:40.517472 19674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:40.517761 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:40.518019 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:40.522970 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.523002 19680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.523236 19674 server_base.cc:1034] running on GCE node
W20260924 17:47:40.522974 19682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.523597 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:40.523916 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:40.525086 19674 hybrid_clock.cc:648] HybridClock initialized: now 1790272060525063 us; error 35 us; skew 500 ppm
I20260924 17:47:40.526495 19674 webserver.cc:460] Webserver started at http://127.1.8.190:46513/ using document root <none> and password file <none>
I20260924 17:47:40.526760 19674 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:40.526850 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:40.527010 19674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:40.528136 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "43fae73773224a2dbd1a694cd9d0f964"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "d433d25accf00ea1272361bb42e83f9b"
server_key_iv: "d63bc7dd7241ba9612d6243dd28037eb"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.528512 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "43fae73773224a2dbd1a694cd9d0f964"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "d433d25accf00ea1272361bb42e83f9b"
server_key_iv: "d63bc7dd7241ba9612d6243dd28037eb"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.529968 19674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:40.530777 19688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.530949 19674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.531059 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "43fae73773224a2dbd1a694cd9d0f964"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "d433d25accf00ea1272361bb42e83f9b"
server_key_iv: "d63bc7dd7241ba9612d6243dd28037eb"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.531141 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:40.550652 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:40.551005 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:40.551232 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:40.555382 19674 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:44421
I20260924 17:47:40.555414 19740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:44421 every 8 connection(s)
I20260924 17:47:40.555781 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:40.556782 19741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:40.558279 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19674
I20260924 17:47:40.558377 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:40.558635 1058 external_mini_cluster.cc:1428] Setting key fe19f870e6da248b0d094b9168c215b1
I20260924 17:47:40.559695 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964: Bootstrap starting.
I20260924 17:47:40.560376 19741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:40.560701 19741 log.cc:824] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:40.561434 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964: No bootstrap required, opened a new log
I20260924 17:47:40.563975 19741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fae73773224a2dbd1a694cd9d0f964" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44421 } }
I20260924 17:47:40.564158 19741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:40.564185 19741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fae73773224a2dbd1a694cd9d0f964, State: Initialized, Role: FOLLOWER
I20260924 17:47:40.564342 19741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fae73773224a2dbd1a694cd9d0f964" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44421 } }
I20260924 17:47:40.564417 19741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:40.564440 19741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:40.564508 19741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:40.565055 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.559231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.1.8.190:44421)
0924 17:47:40.559431 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.559447 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.559473 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:47:40.559956 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.559964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.560031 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.560158 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.560411 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.561603 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.562767 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.563180 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.563274 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.564066 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.564070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.564070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:40.564333 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.564382 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:40.564399 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.564482 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.564750 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.564837 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260924 17:47:40.565280 19741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fae73773224a2dbd1a694cd9d0f964" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44421 } }
I20260924 17:47:40.565368 19741 leader_election.cc:302] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43fae73773224a2dbd1a694cd9d0f964; no voters:
I20260924 17:47:40.565582 19741 leader_election.cc:288] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:40.565613 19746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:40.565722 19746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [term 1 LEADER]: Becoming Leader. State: Replica: 43fae73773224a2dbd1a694cd9d0f964, State: Running, Role: LEADER
I20260924 17:47:40.565866 19741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:40.565868 19746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fae73773224a2dbd1a694cd9d0f964" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44421 } }
I20260924 17:47:40.566265 19746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43fae73773224a2dbd1a694cd9d0f964. Latest consensus state: current_term: 1 leader_uuid: "43fae73773224a2dbd1a694cd9d0f964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fae73773224a2dbd1a694cd9d0f964" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44421 } } }
I20260924 17:47:40.566469 19746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:40.566298 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43fae73773224a2dbd1a694cd9d0f964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43fae73773224a2dbd1a694cd9d0f964" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44421 } } }
I20260924 17:47:40.567276 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:40.567312 19757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:40.568024 19757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:40.569623 19757 catalog_manager.cc:1347] Generated new cluster ID: e21bde36be004b09929e3e0f0311ee46
I20260924 17:47:40.569666 19757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:40.578135 19757 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:40.578820 19757 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:40.580673 19757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964: Generated new TSK 0
I20260924 17:47:40.580857 19757 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:40.585916 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44421
--builtin_ntp_servers=127.1.8.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:40.676882 19765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:40.677100 19765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:40.677124 19765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:40.678478 19765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:40.678536 19765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:40.678565 19765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:40.680011 19765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:40.680267 19765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:40.680596 19765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:40.686738 19773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.686658 19771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.686664 19770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.686983 19765 server_base.cc:1034] running on GCE node
I20260924 17:47:40.687196 19765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:40.687474 19765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:40.688638 19765 hybrid_clock.cc:648] HybridClock initialized: now 1790272060688611 us; error 38 us; skew 500 ppm
I20260924 17:47:40.695370 19765 webserver.cc:460] Webserver started at http://127.1.8.129:43985/ using document root <none> and password file <none>
I20260924 17:47:40.695675 19765 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:40.695729 19765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:40.695875 19765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:40.697031 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "fe538840486e45a3851b1dad730d9dc4"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "d81bc6372729d57312eb3120140b0963"
server_key_iv: "29e1fcb49acd9d265dc5ef5b5882baf0"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.697419 19765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "fe538840486e45a3851b1dad730d9dc4"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "d81bc6372729d57312eb3120140b0963"
server_key_iv: "29e1fcb49acd9d265dc5ef5b5882baf0"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.698894 19765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:40.699736 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.699889 19765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:40.699980 19765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "fe538840486e45a3851b1dad730d9dc4"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "d81bc6372729d57312eb3120140b0963"
server_key_iv: "29e1fcb49acd9d265dc5ef5b5882baf0"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.700052 19765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:40.711748 19765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:40.712091 19765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:40.712253 19765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:40.712491 19765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:40.712827 19765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:40.712883 19765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.712926 19765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:40.712954 19765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.719019 19765 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:44223
I20260924 17:47:40.719061 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:44223 every 8 connection(s)
I20260924 17:47:40.719473 19765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:47:40.721736 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19765
I20260924 17:47:40.721891 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:47:40.722098 1058 external_mini_cluster.cc:1428] Setting key f231ec1d0d03ff5938c11b0a3e212349
I20260924 17:47:40.723165 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44421
--builtin_ntp_servers=127.1.8.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:47:40.725883 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.720262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:54513 (local address 127.1.8.190:44421)
0924 17:47:40.720327 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.720329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.720560 (+ 231us) server_negotiation.cc:407] Connection header received
0924 17:47:40.721045 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.721046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.721059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.721129 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.721836 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.722592 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.724212 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.724410 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.724519 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.725550 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.725551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.725552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:40.725558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.725577 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:40.725582 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.725640 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.725733 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.725762 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 17:47:40.726022 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.720235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44421 (local address 127.1.8.129:54513)
0924 17:47:40.720428 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.720472 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.720961 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.721193 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.721196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.721205 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:40.721710 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.721714 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.722702 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.722703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.724106 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.724110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.724530 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.724531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.724710 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.725397 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:40.725397 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:40.725429 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:40.725454 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:40.725647 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:40.725651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:40.725653 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.725711 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.725799 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260924 17:47:40.726591 19893 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44421
I20260924 17:47:40.726711 19893 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:40.726945 19893 heartbeater.cc:511] Master 127.1.8.190:44421 requested a full tablet report, sending...
I20260924 17:47:40.727494 19705 ts_manager.cc:194] Registered new tserver with Master: fe538840486e45a3851b1dad730d9dc4 (127.1.8.129:44223)
I20260924 17:47:40.728121 19705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.129:54513
W20260924 17:47:40.806124 19896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:40.806344 19896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:40.806370 19896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:40.807622 19896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:40.807668 19896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:40.807696 19896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:40.809084 19896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:40.809350 19896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:40.809666 19896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:40.814785 19904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.814785 19902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.814900 19896 server_base.cc:1034] running on GCE node
W20260924 17:47:40.814857 19901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.815294 19896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:40.815506 19896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:40.816648 19896 hybrid_clock.cc:648] HybridClock initialized: now 1790272060816627 us; error 37 us; skew 500 ppm
I20260924 17:47:40.817924 19896 webserver.cc:460] Webserver started at http://127.1.8.130:44611/ using document root <none> and password file <none>
I20260924 17:47:40.818185 19896 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:40.818243 19896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:40.818445 19896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:40.819581 19896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "fd0189a06ee64ac4b9a14ad7ca21086e"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "c89d7e7ea98eac7d28f098ad4d583db4"
server_key_iv: "0ce7e08ebd804aa1977b7f07dc600104"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.819959 19896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "fd0189a06ee64ac4b9a14ad7ca21086e"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "c89d7e7ea98eac7d28f098ad4d583db4"
server_key_iv: "0ce7e08ebd804aa1977b7f07dc600104"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.821307 19896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:40.822075 19910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.822263 19896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:47:40.822336 19896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "fd0189a06ee64ac4b9a14ad7ca21086e"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "c89d7e7ea98eac7d28f098ad4d583db4"
server_key_iv: "0ce7e08ebd804aa1977b7f07dc600104"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.822460 19896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:40.834093 19896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:40.834503 19896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:40.834661 19896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:40.834914 19896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:40.835273 19896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:40.835307 19896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.835330 19896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:40.835376 19896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.841323 19896 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:38139
I20260924 17:47:40.841374 20023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:38139 every 8 connection(s)
I20260924 17:47:40.841774 19896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:40.847599 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.842654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:50817 (local address 127.1.8.190:44421)
0924 17:47:40.842706 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.842709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.842945 (+ 236us) server_negotiation.cc:407] Connection header received
0924 17:47:40.843534 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.843535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.843557 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.843630 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.844367 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.844893 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.845947 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.846156 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.846255 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.847256 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.847257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.847258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:40.847264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.847280 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:40.847285 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.847361 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.847470 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.847501 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:47:40.847822 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.842612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44421 (local address 127.1.8.130:50817)
0924 17:47:40.842809 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.842845 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.843411 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.843705 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.843709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.843720 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:40.844259 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.844264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.845013 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.845014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.845843 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.845845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.846267 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.846268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.846439 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.847086 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:40.847086 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:40.847117 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:40.847143 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:40.847361 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:40.847367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:40.847370 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.847471 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.847578 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260924 17:47:40.848376 20024 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44421
I20260924 17:47:40.848471 20024 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:40.848721 20024 heartbeater.cc:511] Master 127.1.8.190:44421 requested a full tablet report, sending...
I20260924 17:47:40.848930 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 19896
I20260924 17:47:40.849031 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:47:40.849139 19705 ts_manager.cc:194] Registered new tserver with Master: fd0189a06ee64ac4b9a14ad7ca21086e (127.1.8.130:38139)
I20260924 17:47:40.849296 1058 external_mini_cluster.cc:1428] Setting key e2b7545483a4865702dab2876772179e
I20260924 17:47:40.849588 19705 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.130:50817
I20260924 17:47:40.850543 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44421
--builtin_ntp_servers=127.1.8.148:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:40.935715 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:40.935941 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:40.935966 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:40.937098 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:40.937142 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:40.937170 20027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:40.938519 20027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:44421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:40.938733 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:40.939005 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:40.944455 20035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.944478 20033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:40.944497 20032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:40.944497 20027 server_base.cc:1034] running on GCE node
I20260924 17:47:40.944839 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:40.945065 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:40.946249 20027 hybrid_clock.cc:648] HybridClock initialized: now 1790272060946235 us; error 35 us; skew 500 ppm
I20260924 17:47:40.947624 20027 webserver.cc:460] Webserver started at http://127.1.8.131:42425/ using document root <none> and password file <none>
I20260924 17:47:40.947914 20027 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:40.947991 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:40.948149 20027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:40.949251 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "053062a244d045178db9107812c6babc"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "42813bbda3948fc5a6aa45f1518a97bd"
server_key_iv: "6a8753353f460d8024fd4463c77da20d"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.949659 20027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "053062a244d045178db9107812c6babc"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "42813bbda3948fc5a6aa45f1518a97bd"
server_key_iv: "6a8753353f460d8024fd4463c77da20d"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.951061 20027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 17:47:40.951798 20041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.951969 20027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:47:40.952070 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "053062a244d045178db9107812c6babc"
format_stamp: "Formatted at 2026-09-24 17:47:40 on dist-test-slave-r7hf"
server_key: "42813bbda3948fc5a6aa45f1518a97bd"
server_key_iv: "6a8753353f460d8024fd4463c77da20d"
server_key_version: "encryptionkey@0"
I20260924 17:47:40.952153 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:40.964913 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:40.965268 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:40.965467 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:40.965713 20027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:40.966080 20027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:40.966140 20027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.966183 20027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:40.966212 20027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:40.972117 20027 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:40049
I20260924 17:47:40.972160 20154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:40049 every 8 connection(s)
I20260924 17:47:40.972568 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:47:40.975740 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20027
I20260924 17:47:40.975829 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:47:40.976022 1058 external_mini_cluster.cc:1428] Setting key 68ab119789bea5ef8c806fdb7ba0bd97
I20260924 17:47:40.978305 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.973336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:53297 (local address 127.1.8.190:44421)
0924 17:47:40.973398 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.973401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.973605 (+ 204us) server_negotiation.cc:407] Connection header received
0924 17:47:40.974145 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.974146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.974158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.974214 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.974981 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.975482 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.976898 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.977167 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.977224 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.978078 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.978079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.978080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:40.978083 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.978094 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:40.978097 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.978139 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.978221 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.978241 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 17:47:40.978593 20157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.973300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44421 (local address 127.1.8.131:53297)
0924 17:47:40.973483 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.973518 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.974036 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.974314 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.974317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.974326 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:40.974884 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.974890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.975602 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.975603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.976804 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.976807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.977260 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.977261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:40.977373 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.977949 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:40.977950 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:40.977980 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:40.978003 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:40.978154 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:40.978158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:40.978160 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:47:40.978212 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:47:40.978311 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260924 17:47:40.979130 20155 heartbeater.cc:347] Connected to a master server at 127.1.8.190:44421
I20260924 17:47:40.979250 20155 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:40.979436 20155 heartbeater.cc:511] Master 127.1.8.190:44421 requested a full tablet report, sending...
I20260924 17:47:40.979830 19704 ts_manager.cc:194] Registered new tserver with Master: 053062a244d045178db9107812c6babc (127.1.8.131:40049)
I20260924 17:47:40.980211 19704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.131:53297
I20260924 17:47:40.987811 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:47:40.992297 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.989298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.1.8.190:44421)
0924 17:47:40.989352 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.989354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.989506 (+ 152us) server_negotiation.cc:407] Connection header received
0924 17:47:40.989547 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.989548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.989558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.989608 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:40.989803 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.990287 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.991109 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:40.991294 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:40.991349 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:40.991975 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:40.991976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:40.991976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:40.991981 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:40.991995 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:40.991998 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:40.992072 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:40.992143 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:47:40.992180 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:47:40.994016 19704 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:47:40.996217 19704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:47:41.003366 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.998828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:38139 (local address 127.0.0.1:35352)
0924 17:47:40.999022 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.999032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.999074 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.999661 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.999663 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.999665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.999713 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.999715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.000705 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.000706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.002030 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.002033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.003145 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.003200 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.003203 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:41.003260 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 17:47:41.003290 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":127,"threads_started":1}
I20260924 17:47:41.003710 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.999316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.1.8.130:38139)
0924 17:47:40.999476 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.999480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.999492 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:47:40.999536 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.999537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.999550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.999647 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.999797 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.000578 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.002148 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.002932 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003011 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.003113 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:41.003574 (+ 461us) server_negotiation.cc:299] Negotiation successful
0924 17:47:41.003622 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260924 17:47:41.003707 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.999130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.1.8.129:44223)
0924 17:47:40.999325 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.999329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.999337 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:47:40.999370 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.999372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.999384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.999472 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.999653 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.000821 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.002148 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.002954 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003453 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.003568 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:41.003593 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 17:47:41.003631 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260924 17:47:41.003367 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.998745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44223 (local address 127.0.0.1:53986)
0924 17:47:40.998919 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.998950 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.999043 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.999478 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.999481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.999483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.999563 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.999566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.000923 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.000924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.002062 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.002064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.003046 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.003133 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.003148 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:47:41.003199 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 17:47:41.003261 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":23,"thread_start_us":64,"threads_started":1}
I20260924 17:47:41.004209 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.998828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:40049 (local address 127.0.0.1:57138)
0924 17:47:40.999146 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:40.999154 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:40.999192 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:40.999742 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:40.999743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:40.999745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:40.999779 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:40.999780 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.001028 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.001028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.002922 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.002942 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.003962 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.004037 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.004042 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:47:41.004097 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 17:47:41.004138 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":125,"threads_started":1}
I20260924 17:47:41.004704 19827 tablet_service.cc:1467] Processing CreateTablet for tablet 064426c6742844de820dac80c45bd3ce (DEFAULT_TABLE table=test-table [id=03670e2adc414174867bc4b55369ddc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:41.005106 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064426c6742844de820dac80c45bd3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:41.006840 19958 tablet_service.cc:1467] Processing CreateTablet for tablet 064426c6742844de820dac80c45bd3ce (DEFAULT_TABLE table=test-table [id=03670e2adc414174867bc4b55369ddc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:41.007186 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064426c6742844de820dac80c45bd3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:41.008657 20169 tablet_bootstrap.cc:492] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: Bootstrap starting.
I20260924 17:47:41.009274 20169 tablet_bootstrap.cc:654] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:41.009585 20169 log.cc:824] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:41.010428 20169 tablet_bootstrap.cc:492] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: No bootstrap required, opened a new log
I20260924 17:47:41.010375 20171 tablet_bootstrap.cc:492] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Bootstrap starting.
I20260924 17:47:41.010471 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:40.999115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.1.8.131:40049)
0924 17:47:40.999601 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:40.999605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:40.999614 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:40.999652 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:40.999653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:40.999668 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:40.999748 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:40.999878 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.000945 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003046 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.003836 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.003912 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.004003 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:41.010272 (+ 6269us) server_negotiation.cc:299] Negotiation successful
0924 17:47:41.010337 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":103,"threads_started":1}
I20260924 17:47:41.010500 20169 ts_tablet_manager.cc:1397] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:47:41.011097 20171 tablet_bootstrap.cc:654] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:41.011132 20089 tablet_service.cc:1467] Processing CreateTablet for tablet 064426c6742844de820dac80c45bd3ce (DEFAULT_TABLE table=test-table [id=03670e2adc414174867bc4b55369ddc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:41.011451 20089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064426c6742844de820dac80c45bd3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:41.011467 20171 log.cc:824] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:41.012346 20171 tablet_bootstrap.cc:492] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: No bootstrap required, opened a new log
I20260924 17:47:41.012405 20171 ts_tablet_manager.cc:1397] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 17:47:41.013535 20169 raft_consensus.cc:348] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.013705 20169 raft_consensus.cc:374] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:41.013742 20169 raft_consensus.cc:729] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe538840486e45a3851b1dad730d9dc4, State: Initialized, Role: FOLLOWER
I20260924 17:47:41.013906 20169 consensus_queue.cc:260] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.014045 20173 tablet_bootstrap.cc:492] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Bootstrap starting.
I20260924 17:47:41.014375 20169 ts_tablet_manager.cc:1428] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 17:47:41.014544 19893 heartbeater.cc:503] Master 127.1.8.190:44421 was elected leader, sending a full tablet report...
I20260924 17:47:41.014703 20173 tablet_bootstrap.cc:654] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:41.015040 20173 log.cc:824] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:41.015141 20171 raft_consensus.cc:348] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.015304 20171 raft_consensus.cc:374] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:41.015352 20171 raft_consensus.cc:729] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0189a06ee64ac4b9a14ad7ca21086e, State: Initialized, Role: FOLLOWER
I20260924 17:47:41.015499 20171 consensus_queue.cc:260] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.015820 20171 ts_tablet_manager.cc:1428] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 17:47:41.015853 20024 heartbeater.cc:503] Master 127.1.8.190:44421 was elected leader, sending a full tablet report...
I20260924 17:47:41.015897 20173 tablet_bootstrap.cc:492] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: No bootstrap required, opened a new log
I20260924 17:47:41.015960 20173 ts_tablet_manager.cc:1397] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:47:41.017975 20173 raft_consensus.cc:348] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.018100 20173 raft_consensus.cc:374] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:41.018121 20173 raft_consensus.cc:729] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053062a244d045178db9107812c6babc, State: Initialized, Role: FOLLOWER
I20260924 17:47:41.018234 20173 consensus_queue.cc:260] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.018513 20173 ts_tablet_manager.cc:1428] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:41.018543 20155 heartbeater.cc:503] Master 127.1.8.190:44421 was elected leader, sending a full tablet report...
W20260924 17:47:41.092409 20025 tablet.cc:2369] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:47:41.220252 19894 tablet.cc:2369] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:47:41.223127 20156 tablet.cc:2369] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:47:41.278792 20174 raft_consensus.cc:482] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:41.278942 20174 raft_consensus.cc:504] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.279301 20174 leader_election.cc:288] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd0189a06ee64ac4b9a14ad7ca21086e (127.1.8.130:38139), 053062a244d045178db9107812c6babc (127.1.8.131:40049)
I20260924 17:47:41.282867 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:41.279592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:40049 (local address 127.1.8.129:37315)
0924 17:47:41.279739 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:41.279751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:41.279869 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:41.280046 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:41.280048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:41.280050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:41.280119 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.280121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.280838 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.280839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.281764 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.281767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.282563 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.282564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.282639 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.282643 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:41.282716 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:47:41.282770 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260924 17:47:41.283015 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:41.279663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:52391 (local address 127.1.8.130:38139)
0924 17:47:41.279706 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:41.279708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:41.279820 (+ 112us) server_negotiation.cc:407] Connection header received
0924 17:47:41.279922 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:41.279923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:41.279934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:41.279989 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:41.280186 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.280696 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.281780 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.282339 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.282571 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.282583 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:41.282902 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 17:47:41.282949 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:47:41.283017 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:41.279519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:38139 (local address 127.1.8.129:52391)
0924 17:47:41.279722 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:41.279734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:41.279825 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:41.280005 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:41.280006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:41.280008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:41.280100 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.280102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.280790 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.280792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.281691 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:41.281694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.282611 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.282612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:41.282686 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.282691 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:47:41.282744 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 17:47:41.282787 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":147,"threads_started":1}
I20260924 17:47:41.283592 19978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064426c6742844de820dac80c45bd3ce" candidate_uuid: "fe538840486e45a3851b1dad730d9dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" is_pre_election: true
I20260924 17:47:41.283764 19978 raft_consensus.cc:2454] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe538840486e45a3851b1dad730d9dc4 in term 0.
I20260924 17:47:41.283990 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:41.279592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:37315 (local address 127.1.8.131:40049)
0924 17:47:41.279642 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:41.279644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:41.279863 (+ 219us) server_negotiation.cc:407] Connection header received
0924 17:47:41.279944 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:41.279945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:41.279953 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:41.280028 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:41.280244 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.280712 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.281874 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.282475 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.282547 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.282560 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:41.283873 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0924 17:47:41.283925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:47:41.284017 19781 leader_election.cc:302] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fd0189a06ee64ac4b9a14ad7ca21086e, fe538840486e45a3851b1dad730d9dc4; no voters:
I20260924 17:47:41.284260 20174 raft_consensus.cc:2781] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:47:41.284338 20174 raft_consensus.cc:482] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:47:41.284368 20174 raft_consensus.cc:3037] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:41.284482 20109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064426c6742844de820dac80c45bd3ce" candidate_uuid: "fe538840486e45a3851b1dad730d9dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053062a244d045178db9107812c6babc" is_pre_election: true
I20260924 17:47:41.284621 20109 raft_consensus.cc:2454] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe538840486e45a3851b1dad730d9dc4 in term 0.
I20260924 17:47:41.285228 20174 raft_consensus.cc:504] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.285385 20174 leader_election.cc:288] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [CANDIDATE]: Term 1 election: Requested vote from peers fd0189a06ee64ac4b9a14ad7ca21086e (127.1.8.130:38139), 053062a244d045178db9107812c6babc (127.1.8.131:40049)
I20260924 17:47:41.285531 20109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064426c6742844de820dac80c45bd3ce" candidate_uuid: "fe538840486e45a3851b1dad730d9dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "053062a244d045178db9107812c6babc"
I20260924 17:47:41.285614 20109 raft_consensus.cc:3037] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:41.285597 19978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064426c6742844de820dac80c45bd3ce" candidate_uuid: "fe538840486e45a3851b1dad730d9dc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e"
I20260924 17:47:41.285745 19978 raft_consensus.cc:3037] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:41.286687 20109 raft_consensus.cc:2454] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe538840486e45a3851b1dad730d9dc4 in term 1.
I20260924 17:47:41.286746 19978 raft_consensus.cc:2454] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe538840486e45a3851b1dad730d9dc4 in term 1.
I20260924 17:47:41.286903 19783 leader_election.cc:302] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 053062a244d045178db9107812c6babc, fe538840486e45a3851b1dad730d9dc4; no voters:
I20260924 17:47:41.286990 20174 raft_consensus.cc:2781] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:41.287134 20174 raft_consensus.cc:686] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [term 1 LEADER]: Becoming Leader. State: Replica: fe538840486e45a3851b1dad730d9dc4, State: Running, Role: LEADER
I20260924 17:47:41.287259 20174 consensus_queue.cc:237] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } }
I20260924 17:47:41.288170 19704 catalog_manager.cc:5495] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe538840486e45a3851b1dad730d9dc4 (127.1.8.129). New cstate: current_term: 1 leader_uuid: "fe538840486e45a3851b1dad730d9dc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe538840486e45a3851b1dad730d9dc4" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 44223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 } health_report { overall_health: UNKNOWN } } }
I20260924 17:47:41.361083 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:41.358654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.1.8.129:44223)
0924 17:47:41.358723 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:41.358725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:41.358748 (+ 23us) server_negotiation.cc:407] Connection header received
0924 17:47:41.358783 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:41.358784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:41.358803 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:41.358886 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:47:41.359031 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.359508 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.360306 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:41.360500 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:41.360586 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:41.360744 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:47:41.360838 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:47:41.360951 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:41.360982 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 17:47:41.360993 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:47:41.363112 19978 raft_consensus.cc:1261] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 1 FOLLOWER]: Refusing update from remote peer fe538840486e45a3851b1dad730d9dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:41.363112 20109 raft_consensus.cc:1261] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 1 FOLLOWER]: Refusing update from remote peer fe538840486e45a3851b1dad730d9dc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:41.363543 20180 consensus_queue.cc:1035] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd0189a06ee64ac4b9a14ad7ca21086e" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:41.363687 20174 consensus_queue.cc:1035] T 064426c6742844de820dac80c45bd3ce P fe538840486e45a3851b1dad730d9dc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "053062a244d045178db9107812c6babc" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:41.366074 20183 mvcc.cc:205] Tried to move back new op lower bound from 7332954363340075008 to 7332954363033456640. Current Snapshot: MvccSnapshot[applied={T|T < 7332954363340075008}]
I20260924 17:47:41.366873 20184 mvcc.cc:205] Tried to move back new op lower bound from 7332954363340075008 to 7332954363033456640. Current Snapshot: MvccSnapshot[applied={T|T < 7332954363340075008}]
I20260924 17:47:41.367229 20185 mvcc.cc:205] Tried to move back new op lower bound from 7332954363340075008 to 7332954363033456640. Current Snapshot: MvccSnapshot[applied={T|T < 7332954363340075008 or (T in {7332954363340075008})}]
I20260924 17:47:41.368484 19705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:41.368584 19705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:41.369661 19705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43fae73773224a2dbd1a694cd9d0f964: Sending DeleteTablet for 3 replicas of tablet 064426c6742844de820dac80c45bd3ce
I20260924 17:47:41.370023 19827 tablet_service.cc:1514] Processing DeleteTablet for tablet 064426c6742844de820dac80c45bd3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:41 UTC) from {username='slave'} at 127.0.0.1:53986
I20260924 17:47:41.370040 20089 tablet_service.cc:1514] Processing DeleteTablet for tablet 064426c6742844de820dac80c45bd3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:41 UTC) from {username='slave'} at 127.0.0.1:57138
I20260924 17:47:41.370077 19958 tablet_service.cc:1514] Processing DeleteTablet for tablet 064426c6742844de820dac80c45bd3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:41 UTC) from {username='slave'} at 127.0.0.1:35352
I20260924 17:47:41.370287 20194 tablet_replica.cc:330] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: stopping tablet replica
I20260924 17:47:41.370462 20195 tablet_replica.cc:330] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: stopping tablet replica
I20260924 17:47:41.370514 20194 raft_consensus.cc:2229] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:41.370602 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19765
I20260924 17:47:41.370623 20194 raft_consensus.cc:2258] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:41.371718 20194 ts_tablet_manager.cc:1905] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:41.372889 20195 raft_consensus.cc:2229] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:41.373032 20195 raft_consensus.cc:2258] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:41.373488 20195 ts_tablet_manager.cc:1905] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:41.373780 20194 ts_tablet_manager.cc:1918] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:41.373860 20194 log.cc:1196] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/064426c6742844de820dac80c45bd3ce
I20260924 17:47:41.374116 20194 ts_tablet_manager.cc:1939] T 064426c6742844de820dac80c45bd3ce P 053062a244d045178db9107812c6babc: Deleting consensus metadata
I20260924 17:47:41.374850 19692 catalog_manager.cc:4841] TS 053062a244d045178db9107812c6babc (127.1.8.131:40049): tablet 064426c6742844de820dac80c45bd3ce (table test-table [id=03670e2adc414174867bc4b55369ddc0]) successfully deleted
I20260924 17:47:41.375079 20195 ts_tablet_manager.cc:1918] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:41.375164 20195 log.cc:1196] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/064426c6742844de820dac80c45bd3ce
I20260924 17:47:41.375396 20195 ts_tablet_manager.cc:1939] T 064426c6742844de820dac80c45bd3ce P fd0189a06ee64ac4b9a14ad7ca21086e: Deleting consensus metadata
I20260924 17:47:41.376122 19690 catalog_manager.cc:4841] TS fd0189a06ee64ac4b9a14ad7ca21086e (127.1.8.130:38139): tablet 064426c6742844de820dac80c45bd3ce (table test-table [id=03670e2adc414174867bc4b55369ddc0]) successfully deleted
W20260924 17:47:41.376986 19691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:44223 (error 108)
I20260924 17:47:41.377229 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19896
I20260924 17:47:41.377875 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:41.377354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:44223 (local address 127.0.0.1:54000)
0924 17:47:41.377801 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:41.377849 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:44223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426}
W20260924 17:47:41.381994 19691 catalog_manager.cc:4584] TS fe538840486e45a3851b1dad730d9dc4 (127.1.8.129:44223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 064426c6742844de820dac80c45bd3ce: Network error: Client connection negotiation failed: client connection to 127.1.8.129:44223: connect: Connection refused (error 111)
I20260924 17:47:41.382061 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20027
I20260924 17:47:41.386924 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 19674
2026-09-24T17:47:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (980 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:41 dist-test-slave-r7hf krb5kdc[20199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:41 dist-test-slave-r7hf krb5kdc[20199](info): set up 2 sockets
Sep 24 17:47:41 dist-test-slave-r7hf krb5kdc[20199](info): commencing operation
krb5kdc: starting...
W20260924 17:47:42.422375 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:42 dist-test-slave-r7hf krb5kdc[20199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:47:42.546233 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37429
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:37429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:42.628808 20212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:42.629024 20212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:42.629047 20212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:42.630196 20212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:42.630241 20212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:42.630255 20212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:42.630270 20212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:42.630281 20212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:42.631737 20212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:37429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:42.631939 20212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:42.632124 20212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:42.636983 20220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:42.637118 20217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:42.637133 20218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:42.637372 20212 server_base.cc:1034] running on GCE node
I20260924 17:47:42.637658 20212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:42.637990 20212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:42.639163 20212 hybrid_clock.cc:648] HybridClock initialized: now 1790272062639149 us; error 32 us; skew 500 ppm
Sep 24 17:47:42 dist-test-slave-r7hf krb5kdc[20199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:42.641691 20212 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:47:42.642225 20212 webserver.cc:460] Webserver started at http://127.1.8.190:36881/ using document root <none> and password file <none>
I20260924 17:47:42.642540 20212 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:42.642612 20212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:42.642787 20212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:42.643909 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "b7f6057375aa456cbcd7075d8a4f82c3"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "fc7daf86de748c8bdad3e0eff542b36f"
server_key_iv: "47f99ae14ad5824015530f37b789c7a3"
server_key_version: "encryptionkey@0"
I20260924 17:47:42.644286 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "b7f6057375aa456cbcd7075d8a4f82c3"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "fc7daf86de748c8bdad3e0eff542b36f"
server_key_iv: "47f99ae14ad5824015530f37b789c7a3"
server_key_version: "encryptionkey@0"
I20260924 17:47:42.645696 20212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:42.646528 20227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:42.646723 20212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:42.646838 20212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "b7f6057375aa456cbcd7075d8a4f82c3"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "fc7daf86de748c8bdad3e0eff542b36f"
server_key_iv: "47f99ae14ad5824015530f37b789c7a3"
server_key_version: "encryptionkey@0"
I20260924 17:47:42.646927 20212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:42.662909 20212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:42.664932 20212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:42.665110 20212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:42.669590 20212 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:37429
I20260924 17:47:42.669644 20279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:37429 every 8 connection(s)
I20260924 17:47:42.670033 20212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:42.670979 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:42.671270 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20212
I20260924 17:47:42.671383 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:42.671581 1058 external_mini_cluster.cc:1428] Setting key d65785acf45ea6a1f0f9cac5df689945
I20260924 17:47:42.674705 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3: Bootstrap starting.
I20260924 17:47:42.675459 20280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:42.675773 20280 log.cc:824] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:42.676491 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3: No bootstrap required, opened a new log
I20260924 17:47:42.678519 20280 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37429 } }
I20260924 17:47:42.678735 20280 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:42.678771 20280 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7f6057375aa456cbcd7075d8a4f82c3, State: Initialized, Role: FOLLOWER
Sep 24 17:47:42 dist-test-slave-r7hf krb5kdc[20199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:42.678944 20280 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37429 } }
I20260924 17:47:42.679045 20280 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:42.679095 20280 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:42.679149 20280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:42.679715 20280 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37429 } }
I20260924 17:47:42.679838 20280 leader_election.cc:302] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7f6057375aa456cbcd7075d8a4f82c3; no voters:
I20260924 17:47:42.680016 20280 leader_election.cc:288] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:42.680054 20285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:42.680243 20285 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [term 1 LEADER]: Becoming Leader. State: Replica: b7f6057375aa456cbcd7075d8a4f82c3, State: Running, Role: LEADER
I20260924 17:47:42.680308 20280 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:42.680351 20285 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37429 } }
I20260924 17:47:42.680693 20287 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37429 } } }
I20260924 17:47:42.680683 20285 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7f6057375aa456cbcd7075d8a4f82c3. Latest consensus state: current_term: 1 leader_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7f6057375aa456cbcd7075d8a4f82c3" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 37429 } } }
I20260924 17:47:42.680872 20285 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:42.680861 20287 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:42.681154 20294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:42.681718 20294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:42.683656 20294 catalog_manager.cc:1347] Generated new cluster ID: ae1ac965de704b63b835a583f8fcb240
I20260924 17:47:42.683722 20294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:42.683881 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:42.672142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.1.8.190:37429)
0924 17:47:42.672352 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:42.672362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:42.672400 (+ 38us) server_negotiation.cc:407] Connection header received
0924 17:47:42.672911 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:42.672922 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:42.672991 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:42.673158 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:42.673784 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:42.675208 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:42.676078 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:42.676566 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:42.676712 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:42.679407 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:42.679412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:42.679412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:42.679425 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:42.681739 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:42.682201 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:42.682204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:42.682204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:42.682310 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:42.682596 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:42.682597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:42.682598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:42.683038 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:42.683217 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:42.683503 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 17:47:42.683649 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260924 17:47:42.694491 20294 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:42.695245 20294 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:42.703926 20294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3: Generated new TSK 0
I20260924 17:47:42.704177 20294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.8.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
Entry for principal kudu/127.1.8.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab.
I20260924 17:47:42.735045 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37429
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:42.819751 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:42.820003 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:42.820051 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:42.821269 20306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:42.821344 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:42.821391 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:42.822954 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.129.keytab
--principal=kudu/127.1.8.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:42.823251 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:42.823498 20306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:42.828723 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:42.828739 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:42.828739 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:42.828893 20306 server_base.cc:1034] running on GCE node
I20260924 17:47:42.829258 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:42.829494 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:42.830647 20306 hybrid_clock.cc:648] HybridClock initialized: now 1790272062830625 us; error 46 us; skew 500 ppm
Sep 24 17:47:42 dist-test-slave-r7hf krb5kdc[20199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:42.833415 20306 init.cc:377] Logged in from keytab as kudu/127.1.8.129@KRBTEST.COM (short username kudu)
I20260924 17:47:42.833889 20306 webserver.cc:460] Webserver started at http://127.1.8.129:32927/ using document root <none> and password file <none>
I20260924 17:47:42.834156 20306 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:42.834208 20306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:42.834434 20306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:42.835613 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "28cd273732784ec7ab7081165931361a"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "4814f2e411e7d41186850869d33bed61"
server_key_iv: "9e4162533dcca46bcf22e38b9f5e37d5"
server_key_version: "encryptionkey@0"
I20260924 17:47:42.835997 20306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "28cd273732784ec7ab7081165931361a"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "4814f2e411e7d41186850869d33bed61"
server_key_iv: "9e4162533dcca46bcf22e38b9f5e37d5"
server_key_version: "encryptionkey@0"
I20260924 17:47:42.837502 20306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:42.838446 20321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:42.838650 20306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:42.838752 20306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "28cd273732784ec7ab7081165931361a"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "4814f2e411e7d41186850869d33bed61"
server_key_iv: "9e4162533dcca46bcf22e38b9f5e37d5"
server_key_version: "encryptionkey@0"
I20260924 17:47:42.838812 20306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:42.859413 20306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:42.861847 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:42.861970 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:42.862162 20306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:42.862557 20306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:42.862598 20306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:42.862622 20306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:42.862670 20306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:42.868592 20306 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:46313
I20260924 17:47:42.868678 20434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:46313 every 8 connection(s)
I20260924 17:47:42.869012 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:47:42.870910 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20306
I20260924 17:47:42.871040 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:47:42.871281 1058 external_mini_cluster.cc:1428] Setting key 623ed8ce3bcdfe3bacaf2243f911c74b
Sep 24 17:47:42 dist-test-slave-r7hf krb5kdc[20199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.129@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.130@KRBTEST.COM; defaulting to no policy
I20260924 17:47:42.879776 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:42.869807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:55123 (local address 127.1.8.190:37429)
0924 17:47:42.869872 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:42.869874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:42.870111 (+ 237us) server_negotiation.cc:407] Connection header received
0924 17:47:42.870692 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:42.870693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:42.870703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:42.870758 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:42.872056 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:42.872551 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:42.873880 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:42.874215 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:42.874297 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:42.876836 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:42.876837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:42.876838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:42.876846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:42.878280 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:42.878760 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:42.878762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:42.878763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:42.878825 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:42.879088 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:42.879089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:42.879090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:42.879214 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:42.879308 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:42.879594 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 17:47:42.879678 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 17:47:42.879993 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:42.869796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37429 (local address 127.1.8.129:55123)
0924 17:47:42.869972 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:42.870016 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:42.870595 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:42.870832 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:42.870838 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:42.871693 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:42.871963 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:42.871969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:42.872651 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:42.872652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:42.873783 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:42.873786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:42.874323 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:42.874324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:42.874491 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:42.875028 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:42.875040 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:42.876697 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:42.878414 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:42.878422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:42.878425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:42.878662 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:42.878893 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:42.878894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:42.878894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:42.879011 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:42.879298 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:42.879305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:42.879456 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 17:47:42.879562 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 17:47:42.879748 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260924 17:47:42.880565 20435 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37429
I20260924 17:47:42.880684 20435 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:42.880955 20435 heartbeater.cc:511] Master 127.1.8.190:37429 requested a full tablet report, sending...
I20260924 17:47:42.881551 20244 ts_manager.cc:194] Registered new tserver with Master: 28cd273732784ec7ab7081165931361a (127.1.8.129:46313)
I20260924 17:47:42.882283 20244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.129@KRBTEST.COM'} at 127.1.8.129:55123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
Entry for principal kudu/127.1.8.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab.
I20260924 17:47:42.899705 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37429
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:42.983647 20440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:42.983882 20440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:42.983913 20440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:42.985153 20440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:42.985203 20440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:42.985231 20440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:42.986747 20440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.130.keytab
--principal=kudu/127.1.8.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:42.986971 20440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:42.987165 20440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:42.992622 20445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:42.992697 20440 server_base.cc:1034] running on GCE node
W20260924 17:47:42.992630 20448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:42.992630 20446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:42.993121 20440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:42.993413 20440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:42.994635 20440 hybrid_clock.cc:648] HybridClock initialized: now 1790272062994620 us; error 29 us; skew 500 ppm
Sep 24 17:47:42 dist-test-slave-r7hf krb5kdc[20199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:42.997418 20440 init.cc:377] Logged in from keytab as kudu/127.1.8.130@KRBTEST.COM (short username kudu)
I20260924 17:47:42.998009 20440 webserver.cc:460] Webserver started at http://127.1.8.130:37191/ using document root <none> and password file <none>
I20260924 17:47:42.998270 20440 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:42.998344 20440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:42.998528 20440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:42.999745 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "400c0d1dfb204d37947bfeb8ab14f5dc"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "5972f68b7b384a9875fb3f3f308be740"
server_key_iv: "03e19758c600794dcf1eede44d82e8c1"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.000164 20440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "400c0d1dfb204d37947bfeb8ab14f5dc"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "5972f68b7b384a9875fb3f3f308be740"
server_key_iv: "03e19758c600794dcf1eede44d82e8c1"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.001699 20440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:43.002638 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.002840 20440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:43.002957 20440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "400c0d1dfb204d37947bfeb8ab14f5dc"
format_stamp: "Formatted at 2026-09-24 17:47:42 on dist-test-slave-r7hf"
server_key: "5972f68b7b384a9875fb3f3f308be740"
server_key_iv: "03e19758c600794dcf1eede44d82e8c1"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.003039 20440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:43.019407 20440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:43.021993 20440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:43.022184 20440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:43.022518 20440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:43.022912 20440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:43.022971 20440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.023021 20440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:43.023051 20440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.029302 20440 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:44521
I20260924 17:47:43.029343 20568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:44521 every 8 connection(s)
I20260924 17:47:43.029714 20440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:43.034930 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20440
I20260924 17:47:43.035051 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:47:43.035262 1058 external_mini_cluster.cc:1428] Setting key 7358dca1511260b25fd115151aa1cd6a
Sep 24 17:47:43 dist-test-slave-r7hf krb5kdc[20199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272062, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.130@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:43.040035 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.030589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:37785 (local address 127.1.8.190:37429)
0924 17:47:43.030648 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.030651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.030921 (+ 270us) server_negotiation.cc:407] Connection header received
0924 17:47:43.031418 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.031420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.031430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.031485 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:43.032437 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.032925 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.034056 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.034245 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.034305 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.036830 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:43.036831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:43.036831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:43.036840 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:43.038610 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:43.039009 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:43.039010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:43.039010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:43.039076 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:43.039329 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:43.039330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:43.039330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:43.039489 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:43.039560 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.039819 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.039923 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:47:43.040192 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.030588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37429 (local address 127.1.8.130:37785)
0924 17:47:43.030777 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.030817 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.031325 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.031545 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.031548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.032139 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:43.032352 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.032356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.033009 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.033010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.033969 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.033971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.034321 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.034322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.034494 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.035009 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:43.035021 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:43.036718 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:43.038683 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:43.038689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:43.038692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:43.038930 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:43.039146 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:43.039147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:43.039147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:43.039262 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:43.039554 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:43.039558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:43.039679 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.039765 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.039894 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260924 17:47:43.040787 20569 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37429
I20260924 17:47:43.040874 20569 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:43.041072 20569 heartbeater.cc:511] Master 127.1.8.190:37429 requested a full tablet report, sending...
I20260924 17:47:43.041527 20244 ts_manager.cc:194] Registered new tserver with Master: 400c0d1dfb204d37947bfeb8ab14f5dc (127.1.8.130:44521)
I20260924 17:47:43.041972 20244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.130@KRBTEST.COM'} at 127.1.8.130:37785
WARNING: no policy specified for kudu/127.1.8.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
Entry for principal kudu/127.1.8.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab.
I20260924 17:47:43.064999 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37429
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:35679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:43.147712 20574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:43.148000 20574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:43.148043 20574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:43.149304 20574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:43.149390 20574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:43.149441 20574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:43.151213 20574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:35679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.131.keytab
--principal=kudu/127.1.8.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:43.151510 20574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:43.151767 20574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:43.157099 20580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:43.157092 20582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:43.157092 20579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:43.157408 20574 server_base.cc:1034] running on GCE node
I20260924 17:47:43.157631 20574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:43.157881 20574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:43.159019 20574 hybrid_clock.cc:648] HybridClock initialized: now 1790272063159002 us; error 40 us; skew 500 ppm
Sep 24 17:47:43 dist-test-slave-r7hf krb5kdc[20199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:43.161676 20574 init.cc:377] Logged in from keytab as kudu/127.1.8.131@KRBTEST.COM (short username kudu)
I20260924 17:47:43.162165 20574 webserver.cc:460] Webserver started at http://127.1.8.131:39891/ using document root <none> and password file <none>
I20260924 17:47:43.162489 20574 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:43.162540 20574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:43.162700 20574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:43.164057 20574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "bd1dcc5fa093485bad4e0d487588888b"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "1353238fdfaa4022ec157d15f8b7d898"
server_key_iv: "8c91481dd620d354bc930c9149bbd4df"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.164551 20574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "bd1dcc5fa093485bad4e0d487588888b"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "1353238fdfaa4022ec157d15f8b7d898"
server_key_iv: "8c91481dd620d354bc930c9149bbd4df"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.165963 20574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:43.166832 20589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.166999 20574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.167107 20574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "bd1dcc5fa093485bad4e0d487588888b"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "1353238fdfaa4022ec157d15f8b7d898"
server_key_iv: "8c91481dd620d354bc930c9149bbd4df"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.167199 20574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:43.180825 20574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:43.183449 20574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:43.183620 20574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:43.183913 20574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:43.184252 20574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:43.184310 20574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.184352 20574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:43.184397 20574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.190438 20574 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:40491
I20260924 17:47:43.190480 20702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:40491 every 8 connection(s)
I20260924 17:47:43.190861 20574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
Sep 24 17:47:43 dist-test-slave-r7hf krb5kdc[20199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272063, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.131@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:47:43.200078 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20574
I20260924 17:47:43.200188 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:47:43.200336 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.191643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:52441 (local address 127.1.8.190:37429)
0924 17:47:43.191690 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.191692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.191911 (+ 219us) server_negotiation.cc:407] Connection header received
0924 17:47:43.192408 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.192409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.192419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.192476 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:43.193477 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.193972 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.195066 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.195269 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.195330 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.197488 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:43.197490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:43.197490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:43.197498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:43.199021 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:43.199419 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:43.199420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:43.199420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:43.199471 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:43.199685 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:43.199686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:43.199686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:43.199807 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:43.199857 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.200146 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.200237 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:47:43.200362 1058 external_mini_cluster.cc:1428] Setting key 397909a5f5806a08c63f573fd29df2b2
I20260924 17:47:43.200590 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.191630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:37429 (local address 127.1.8.131:52441)
0924 17:47:43.191784 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.191821 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.192318 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.192540 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.192543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.193167 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:43.193385 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.193389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.194059 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.194060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.194979 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.194982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.195350 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.195351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.195467 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.195955 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:47:43.195965 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:43.197391 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:43.199098 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:43.199105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:43.199107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:43.199347 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:43.199529 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:47:43.199529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:47:43.199530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:47:43.199631 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:47:43.199864 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:43.199868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:43.199995 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.200118 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.200304 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 17:47:43.201233 20703 heartbeater.cc:347] Connected to a master server at 127.1.8.190:37429
I20260924 17:47:43.201341 20703 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:43.201537 20703 heartbeater.cc:511] Master 127.1.8.190:37429 requested a full tablet report, sending...
I20260924 17:47:43.201987 20244 ts_manager.cc:194] Registered new tserver with Master: bd1dcc5fa093485bad4e0d487588888b (127.1.8.131:40491)
I20260924 17:47:43.202342 20244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.8.131@KRBTEST.COM'} at 127.1.8.131:52441
I20260924 17:47:43.212584 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:47:43.220911 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.214033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.1.8.190:37429)
0924 17:47:43.214113 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.214115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.214285 (+ 170us) server_negotiation.cc:407] Connection header received
0924 17:47:43.214362 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.214363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.214372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.214484 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:43.215251 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.215739 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.216671 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.216855 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.216907 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.218174 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:43.218175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:43.218175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:43.218180 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:43.219727 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:43.220095 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:43.220096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:43.220097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:43.220148 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:43.220336 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:43.220337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:43.220337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:43.220485 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:43.220565 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.220732 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.220800 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 17:47:43.222649 20244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:47:43.224255 20244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:47:43.231361 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.226871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:44521 (local address 127.0.0.1:57292)
0924 17:47:43.227282 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.227291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.227330 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.227726 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.227729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.227731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:43.227809 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.227811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.229135 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.229137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.230307 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.230309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.231030 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.231031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.231114 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.231127 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.231195 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.231254 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
I20260924 17:47:43.231480 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.227138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.1.8.131:40491)
0924 17:47:43.227580 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.227584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.227594 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:47:43.227629 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.227631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.227649 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.227721 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:43.227954 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.228768 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.230301 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.231066 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.231184 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.231249 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.231330 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.231377 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":87,"threads_started":1}
I20260924 17:47:43.231828 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.227158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.1.8.129:46313)
0924 17:47:43.228015 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.228020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.228029 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:43.228067 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.228069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.228086 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.228168 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:43.228339 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.229334 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.230453 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.231137 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.231201 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.231249 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.231692 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.231740 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":78,"threads_started":1}
I20260924 17:47:43.232096 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.227356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.1.8.130:44521)
0924 17:47:43.227536 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.227542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.227553 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:47:43.227599 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.227602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.227619 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.227710 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:43.227883 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.229027 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.230430 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.230934 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.231030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.231096 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.231832 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.232024 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260924 17:47:43.231528 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.226854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:40491 (local address 127.0.0.1:45140)
0924 17:47:43.227035 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.227077 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.227148 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.227799 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.227800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.227803 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:43.227863 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.227865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.228873 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.228874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.230187 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.230191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.231187 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.231188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.231253 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.231256 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.231315 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.231350 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":5,"thread_start_us":50,"threads_started":1}
I20260924 17:47:43.231626 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.226866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:46313 (local address 127.0.0.1:52708)
0924 17:47:43.227140 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.227152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.227210 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.228185 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.228186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.228188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:43.228255 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.228257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.229435 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.229436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.230335 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.230337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.231392 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.231393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.231458 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.231462 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.231514 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.231556 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260924 17:47:43.232915 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2255b4e7b2470e838bd8e16571bff5 (DEFAULT_TABLE table=test-table [id=e5f1344024ea4144accb1e24efbbfba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:43.232913 20637 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2255b4e7b2470e838bd8e16571bff5 (DEFAULT_TABLE table=test-table [id=e5f1344024ea4144accb1e24efbbfba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:43.233263 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2255b4e7b2470e838bd8e16571bff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:43.233264 20637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2255b4e7b2470e838bd8e16571bff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:43.234928 20369 tablet_service.cc:1467] Processing CreateTablet for tablet 2d2255b4e7b2470e838bd8e16571bff5 (DEFAULT_TABLE table=test-table [id=e5f1344024ea4144accb1e24efbbfba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:43.235306 20369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d2255b4e7b2470e838bd8e16571bff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:43.236284 20720 tablet_bootstrap.cc:492] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Bootstrap starting.
I20260924 17:47:43.236688 20721 tablet_bootstrap.cc:492] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Bootstrap starting.
I20260924 17:47:43.237174 20720 tablet_bootstrap.cc:654] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:43.237373 20721 tablet_bootstrap.cc:654] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:43.237836 20720 log.cc:824] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:43.237982 20721 log.cc:824] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:43.238576 20724 tablet_bootstrap.cc:492] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: Bootstrap starting.
I20260924 17:47:43.238729 20720 tablet_bootstrap.cc:492] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: No bootstrap required, opened a new log
I20260924 17:47:43.238829 20720 ts_tablet_manager.cc:1397] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:47:43.239346 20724 tablet_bootstrap.cc:654] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:43.239362 20721 tablet_bootstrap.cc:492] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: No bootstrap required, opened a new log
I20260924 17:47:43.239424 20721 ts_tablet_manager.cc:1397] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 17:47:43.239974 20724 log.cc:824] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:43.240636 20720 raft_consensus.cc:348] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.240811 20720 raft_consensus.cc:374] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:43.240825 20724 tablet_bootstrap.cc:492] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: No bootstrap required, opened a new log
I20260924 17:47:43.240854 20720 raft_consensus.cc:729] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1dcc5fa093485bad4e0d487588888b, State: Initialized, Role: FOLLOWER
I20260924 17:47:43.240885 20724 ts_tablet_manager.cc:1397] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 17:47:43.240972 20720 consensus_queue.cc:260] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.241307 20720 ts_tablet_manager.cc:1428] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 17:47:43.241328 20703 heartbeater.cc:503] Master 127.1.8.190:37429 was elected leader, sending a full tablet report...
I20260924 17:47:43.241735 20721 raft_consensus.cc:348] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.241935 20721 raft_consensus.cc:374] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:43.241971 20721 raft_consensus.cc:729] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400c0d1dfb204d37947bfeb8ab14f5dc, State: Initialized, Role: FOLLOWER
I20260924 17:47:43.242086 20721 consensus_queue.cc:260] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.242381 20721 ts_tablet_manager.cc:1428] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:43.242424 20569 heartbeater.cc:503] Master 127.1.8.190:37429 was elected leader, sending a full tablet report...
I20260924 17:47:43.243573 20724 raft_consensus.cc:348] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.243719 20724 raft_consensus.cc:374] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:43.243741 20724 raft_consensus.cc:729] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cd273732784ec7ab7081165931361a, State: Initialized, Role: FOLLOWER
I20260924 17:47:43.243891 20724 consensus_queue.cc:260] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.244197 20724 ts_tablet_manager.cc:1428] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:43.244203 20435 heartbeater.cc:503] Master 127.1.8.190:37429 was elected leader, sending a full tablet report...
W20260924 17:47:43.280402 20570 tablet.cc:2369] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:47:43.369803 20436 tablet.cc:2369] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 17:47:43.441480 20704 tablet.cc:2369] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:47:43.547582 20726 raft_consensus.cc:482] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:43.547720 20726 raft_consensus.cc:504] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.548203 20726 leader_election.cc:288] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 400c0d1dfb204d37947bfeb8ab14f5dc (127.1.8.130:44521), 28cd273732784ec7ab7081165931361a (127.1.8.129:46313)
I20260924 17:47:43.552270 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.548484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:44219 (local address 127.1.8.130:44521)
0924 17:47:43.548530 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.548532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.548543 (+ 11us) server_negotiation.cc:407] Connection header received
0924 17:47:43.548639 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.548640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.548654 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.548725 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:43.548929 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.549709 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.551338 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.551899 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.551974 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.551985 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.552157 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.552196 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:47:43.552271 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.548562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:33107 (local address 127.1.8.129:46313)
0924 17:47:43.548637 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.548639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.548655 (+ 16us) server_negotiation.cc:407] Connection header received
0924 17:47:43.548796 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.548797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.548809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.548864 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:43.549018 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.549716 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.551338 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.551890 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.551974 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.551985 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.552158 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.552196 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:47:43.552280 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.548433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:44521 (local address 127.1.8.131:44219)
0924 17:47:43.548495 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.548507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.548556 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.548762 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.548763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.548765 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:43.548834 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.548836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.549811 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.549811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.551236 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.551239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.551989 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.551989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.552068 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.552072 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.552132 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.552175 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":82}
I20260924 17:47:43.552292 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.548445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:46313 (local address 127.1.8.131:33107)
0924 17:47:43.548607 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.548616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.548668 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.548888 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.548889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.548890 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:43.548947 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.548949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.549811 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.549811 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.551249 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.551250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.551989 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.551989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.552068 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.552072 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.552132 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.552175 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":78,"thread_start_us":114,"threads_started":1}
I20260924 17:47:43.552647 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2255b4e7b2470e838bd8e16571bff5" candidate_uuid: "bd1dcc5fa093485bad4e0d487588888b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" is_pre_election: true
I20260924 17:47:43.552668 20389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2255b4e7b2470e838bd8e16571bff5" candidate_uuid: "bd1dcc5fa093485bad4e0d487588888b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cd273732784ec7ab7081165931361a" is_pre_election: true
I20260924 17:47:43.552829 20523 raft_consensus.cc:2454] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1dcc5fa093485bad4e0d487588888b in term 0.
I20260924 17:47:43.552845 20389 raft_consensus.cc:2454] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd1dcc5fa093485bad4e0d487588888b in term 0.
I20260924 17:47:43.553078 20593 leader_election.cc:302] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cd273732784ec7ab7081165931361a, bd1dcc5fa093485bad4e0d487588888b; no voters:
I20260924 17:47:43.553222 20726 raft_consensus.cc:2781] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:47:43.553305 20726 raft_consensus.cc:482] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:47:43.553340 20726 raft_consensus.cc:3037] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:43.554055 20726 raft_consensus.cc:504] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.554207 20726 leader_election.cc:288] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [CANDIDATE]: Term 1 election: Requested vote from peers 400c0d1dfb204d37947bfeb8ab14f5dc (127.1.8.130:44521), 28cd273732784ec7ab7081165931361a (127.1.8.129:46313)
I20260924 17:47:43.554428 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2255b4e7b2470e838bd8e16571bff5" candidate_uuid: "bd1dcc5fa093485bad4e0d487588888b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc"
I20260924 17:47:43.554388 20389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d2255b4e7b2470e838bd8e16571bff5" candidate_uuid: "bd1dcc5fa093485bad4e0d487588888b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cd273732784ec7ab7081165931361a"
I20260924 17:47:43.554502 20523 raft_consensus.cc:3037] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:43.554502 20389 raft_consensus.cc:3037] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:43.555368 20389 raft_consensus.cc:2454] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1dcc5fa093485bad4e0d487588888b in term 1.
I20260924 17:47:43.555368 20523 raft_consensus.cc:2454] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd1dcc5fa093485bad4e0d487588888b in term 1.
I20260924 17:47:43.555572 20593 leader_election.cc:302] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cd273732784ec7ab7081165931361a, bd1dcc5fa093485bad4e0d487588888b; no voters:
I20260924 17:47:43.555696 20726 raft_consensus.cc:2781] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:43.555858 20726 raft_consensus.cc:686] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 1 LEADER]: Becoming Leader. State: Replica: bd1dcc5fa093485bad4e0d487588888b, State: Running, Role: LEADER
I20260924 17:47:43.555954 20726 consensus_queue.cc:237] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } }
I20260924 17:47:43.556797 20243 catalog_manager.cc:5495] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b reported cstate change: term changed from 0 to 1, leader changed from <none> to bd1dcc5fa093485bad4e0d487588888b (127.1.8.131). New cstate: current_term: 1 leader_uuid: "bd1dcc5fa093485bad4e0d487588888b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd1dcc5fa093485bad4e0d487588888b" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 40491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 } health_report { overall_health: UNKNOWN } } }
I20260924 17:47:43.584664 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.582093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.1.8.131:40491)
0924 17:47:43.582162 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.582164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.582174 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:47:43.582245 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.582246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.582267 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.582344 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:47:43.582539 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.583031 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.583825 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.584035 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.584123 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.584297 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:47:43.584411 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:47:43.584509 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.584527 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.584547 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 17:47:43.586609 20523 raft_consensus.cc:1261] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 1 FOLLOWER]: Refusing update from remote peer bd1dcc5fa093485bad4e0d487588888b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:43.586609 20389 raft_consensus.cc:1261] T 2d2255b4e7b2470e838bd8e16571bff5 P 28cd273732784ec7ab7081165931361a [term 1 FOLLOWER]: Refusing update from remote peer bd1dcc5fa093485bad4e0d487588888b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:43.586969 20726 consensus_queue.cc:1035] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28cd273732784ec7ab7081165931361a" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:43.587123 20730 consensus_queue.cc:1035] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [LEADER]: Connected to new peer: Peer: permanent_uuid: "400c0d1dfb204d37947bfeb8ab14f5dc" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:43.590365 20735 mvcc.cc:205] Tried to move back new op lower bound from 7332954372447498240 to 7332954372325666816. Current Snapshot: MvccSnapshot[applied={T|T < 7332954372447498240}]
I20260924 17:47:43.590857 20733 mvcc.cc:205] Tried to move back new op lower bound from 7332954372447498240 to 7332954372325666816. Current Snapshot: MvccSnapshot[applied={T|T < 7332954372447498240 or (T in {7332954372447498240})}]
I20260924 17:47:43.592133 20243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:43.592232 20243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:43.593052 20243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7f6057375aa456cbcd7075d8a4f82c3: Sending DeleteTablet for 3 replicas of tablet 2d2255b4e7b2470e838bd8e16571bff5
I20260924 17:47:43.593415 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2255b4e7b2470e838bd8e16571bff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:43 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:57292
I20260924 17:47:43.593492 20637 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2255b4e7b2470e838bd8e16571bff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:43 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:45140
I20260924 17:47:43.593624 20369 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d2255b4e7b2470e838bd8e16571bff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:43 UTC) from {username='kudu', principal='kudu/127.1.8.190@KRBTEST.COM'} at 127.0.0.1:52708
I20260924 17:47:43.593729 20743 tablet_replica.cc:330] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: stopping tablet replica
I20260924 17:47:43.593721 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20306
I20260924 17:47:43.594691 20742 tablet_replica.cc:330] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: stopping tablet replica
I20260924 17:47:43.593842 20743 raft_consensus.cc:2229] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:47:43.594822 20743 raft_consensus.cc:2258] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:43.594872 20742 raft_consensus.cc:2229] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:43.595219 20742 raft_consensus.cc:2258] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:43.595338 20743 ts_tablet_manager.cc:1905] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:43.595746 20742 ts_tablet_manager.cc:1905] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:43.597765 20743 ts_tablet_manager.cc:1918] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:43.597849 20743 log.cc:1196] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/2d2255b4e7b2470e838bd8e16571bff5
I20260924 17:47:43.598052 20742 ts_tablet_manager.cc:1918] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:43.598125 20743 ts_tablet_manager.cc:1939] T 2d2255b4e7b2470e838bd8e16571bff5 P bd1dcc5fa093485bad4e0d487588888b: Deleting consensus metadata
I20260924 17:47:43.598184 20742 log.cc:1196] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/2d2255b4e7b2470e838bd8e16571bff5
I20260924 17:47:43.598531 20742 ts_tablet_manager.cc:1939] T 2d2255b4e7b2470e838bd8e16571bff5 P 400c0d1dfb204d37947bfeb8ab14f5dc: Deleting consensus metadata
I20260924 17:47:43.598785 20230 catalog_manager.cc:4841] TS bd1dcc5fa093485bad4e0d487588888b (127.1.8.131:40491): tablet 2d2255b4e7b2470e838bd8e16571bff5 (table test-table [id=e5f1344024ea4144accb1e24efbbfba7]) successfully deleted
I20260924 17:47:43.598973 20229 catalog_manager.cc:4841] TS 400c0d1dfb204d37947bfeb8ab14f5dc (127.1.8.130:44521): tablet 2d2255b4e7b2470e838bd8e16571bff5 (table test-table [id=e5f1344024ea4144accb1e24efbbfba7]) successfully deleted
W20260924 17:47:43.601006 20231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:46313 (error 108)
I20260924 17:47:43.601200 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20440
I20260924 17:47:43.603132 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.602962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:46313 (local address 127.0.0.1:52722)
0924 17:47:43.603037 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.603092 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 17:47:43.603276 20231 catalog_manager.cc:4584] TS 28cd273732784ec7ab7081165931361a (127.1.8.129:46313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d2255b4e7b2470e838bd8e16571bff5: Network error: Client connection negotiation failed: client connection to 127.1.8.129:46313: connect: Connection refused (error 111)
I20260924 17:47:43.606225 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20574
I20260924 17:47:43.611137 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20212
2026-09-24T17:47:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2225 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T17:47:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:43Z Disabled control of system clock
I20260924 17:47:43.639045 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41873
--webserver_interface=127.1.8.190
--webserver_port=0
--builtin_ntp_servers=127.1.8.148:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:41873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:43.722007 20749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:43.722265 20749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:43.722287 20749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:43.723587 20749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:43.723634 20749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:43.723646 20749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:43.723659 20749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:43.723672 20749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:43.725006 20749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:41873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:43.725237 20749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:43.725500 20749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:43.730542 20754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:43.730575 20755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:43.730672 20749 server_base.cc:1034] running on GCE node
W20260924 17:47:43.730526 20757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:43.731155 20749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:43.731484 20749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:43.732659 20749 hybrid_clock.cc:648] HybridClock initialized: now 1790272063732646 us; error 45 us; skew 500 ppm
I20260924 17:47:43.734062 20749 webserver.cc:460] Webserver started at http://127.1.8.190:40285/ using document root <none> and password file <none>
I20260924 17:47:43.734370 20749 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:43.734500 20749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:43.734660 20749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:43.735775 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "dc9c5252fc4f4d4eb3399385b145cb75"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "89050cfc32ab0e4707e37724d23531a7"
server_key_iv: "629fd7200dd8fb457f958b8e197decc1"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.736179 20749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "dc9c5252fc4f4d4eb3399385b145cb75"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "89050cfc32ab0e4707e37724d23531a7"
server_key_iv: "629fd7200dd8fb457f958b8e197decc1"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.737572 20749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:43.738385 20763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.738616 20749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:43.738722 20749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "dc9c5252fc4f4d4eb3399385b145cb75"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "89050cfc32ab0e4707e37724d23531a7"
server_key_iv: "629fd7200dd8fb457f958b8e197decc1"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.738819 20749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:43.755738 20749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:43.756137 20749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:43.756315 20749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:43.760310 20749 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:41873
I20260924 17:47:43.760345 20815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:41873 every 8 connection(s)
I20260924 17:47:43.760717 20749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:43.761785 20816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:43.763796 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20749
I20260924 17:47:43.763916 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:43.764127 1058 external_mini_cluster.cc:1428] Setting key a32f26d61881246d2dc95d0ef81f1b8d
I20260924 17:47:43.764410 20816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: Bootstrap starting.
I20260924 17:47:43.765118 20816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:43.765436 20816 log.cc:824] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:43.766189 20816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: No bootstrap required, opened a new log
I20260924 17:47:43.768388 20816 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41873 } }
I20260924 17:47:43.768544 20816 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:43.768620 20816 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9c5252fc4f4d4eb3399385b145cb75, State: Initialized, Role: FOLLOWER
I20260924 17:47:43.768810 20816 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41873 } }
I20260924 17:47:43.768911 20816 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:43.768937 20816 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:43.769007 20816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:43.769692 20816 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41873 } }
I20260924 17:47:43.769814 20816 leader_election.cc:302] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9c5252fc4f4d4eb3399385b145cb75; no voters:
I20260924 17:47:43.770008 20816 leader_election.cc:288] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:43.770043 20821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:43.770074 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.764714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.1.8.190:41873)
0924 17:47:43.764936 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.764945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.764969 (+ 24us) server_negotiation.cc:407] Connection header received
0924 17:47:43.765450 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.765457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.765566 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.765729 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:43.765943 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.766942 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.768086 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.768488 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.768634 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.769365 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:43.769369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:43.769369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:43.769538 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:43.769570 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:43.769580 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:43.769634 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.769737 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.769839 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260924 17:47:43.770161 20821 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [term 1 LEADER]: Becoming Leader. State: Replica: dc9c5252fc4f4d4eb3399385b145cb75, State: Running, Role: LEADER
I20260924 17:47:43.770292 20816 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:43.770278 20821 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41873 } }
I20260924 17:47:43.770623 20821 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9c5252fc4f4d4eb3399385b145cb75. Latest consensus state: current_term: 1 leader_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41873 } } }
I20260924 17:47:43.770637 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9c5252fc4f4d4eb3399385b145cb75" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 41873 } } }
I20260924 17:47:43.770709 20821 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:43.770941 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75 [sys.catalog]: This master's current role is: LEADER
W20260924 17:47:43.772024 20836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:47:43.772075 20836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:47:43.772131 20827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:43.772306 20827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:43.773844 20827 catalog_manager.cc:1347] Generated new cluster ID: 270345dda6234d72be8a9a55580fc1dc
I20260924 17:47:43.773898 20827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:43.784862 20827 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:43.785494 20827 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:43.797376 20827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: Generated new TSK 0
I20260924 17:47:43.797633 20827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:43.801344 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.129:0
--local_ip_for_outbound_sockets=127.1.8.129
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41873
--builtin_ntp_servers=127.1.8.148:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 17:47:43.887454 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:43.887676 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:43.887699 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:43.889005 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:43.889060 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:43.889088 20840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.129
I20260924 17:47:43.890631 20840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.8.129
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.129
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:43.890908 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:43.891242 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:43.897380 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:43.897457 20840 server_base.cc:1034] running on GCE node
W20260924 17:47:43.897395 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:43.897384 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:43.897760 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:43.898031 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:43.899196 20840 hybrid_clock.cc:648] HybridClock initialized: now 1790272063899190 us; error 39 us; skew 500 ppm
I20260924 17:47:43.900514 20840 webserver.cc:460] Webserver started at http://127.1.8.129:41965/ using document root <none> and password file <none>
I20260924 17:47:43.900792 20840 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:43.900868 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:43.901021 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:43.902169 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data/instance:
uuid: "ce1d7a67de754376b1b67083651ce231"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "bb1345a1f29ceaa6a7f9d8b4acc892e6"
server_key_iv: "6ca23cbcba9956726e6eb08174b5ecbd"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.902575 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance:
uuid: "ce1d7a67de754376b1b67083651ce231"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "bb1345a1f29ceaa6a7f9d8b4acc892e6"
server_key_iv: "6ca23cbcba9956726e6eb08174b5ecbd"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.903963 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:43.904733 20854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.904927 20840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:47:43.905030 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal
uuid: "ce1d7a67de754376b1b67083651ce231"
format_stamp: "Formatted at 2026-09-24 17:47:43 on dist-test-slave-r7hf"
server_key: "bb1345a1f29ceaa6a7f9d8b4acc892e6"
server_key_iv: "6ca23cbcba9956726e6eb08174b5ecbd"
server_key_version: "encryptionkey@0"
I20260924 17:47:43.905117 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:43.922789 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:43.923157 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:43.923331 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:43.923588 20840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:43.923938 20840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:43.923995 20840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.924036 20840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:43.924067 20840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:43.930045 20840 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.129:38187
I20260924 17:47:43.930119 20967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.129:38187 every 8 connection(s)
I20260924 17:47:43.930553 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/data/info.pb
I20260924 17:47:43.936210 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.931429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.129:44623 (local address 127.1.8.190:41873)
0924 17:47:43.931506 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:43.931509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:43.931708 (+ 199us) server_negotiation.cc:407] Connection header received
0924 17:47:43.932288 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:43.932289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:43.932302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:43.932359 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:43.933087 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.933625 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.934730 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.934940 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.935017 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.935920 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:43.935922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:43.935922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:43.935928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:43.935942 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:43.935947 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:43.936000 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:43.936095 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 17:47:43.936128 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:47:43.936420 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:43.931381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41873 (local address 127.1.8.129:44623)
0924 17:47:43.931578 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:43.931623 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:43.932172 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:43.932431 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:43.932435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:43.932445 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:43.932989 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.932993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.933730 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.933731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.934630 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:43.934633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:43.935024 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:43.935026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:43.935164 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:43.935749 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:43.935755 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:43.935785 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:43.935810 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:43.936002 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:43.936007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:43.936010 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:43.936098 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 17:47:43.936202 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260924 17:47:43.937023 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20840
I20260924 17:47:43.937053 20968 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41873
I20260924 17:47:43.937142 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-0/wal/instance
I20260924 17:47:43.937165 20968 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:43.937364 1058 external_mini_cluster.cc:1428] Setting key 91396f8bd8b6c08c8dd3f29e86e2b8cc
I20260924 17:47:43.937367 20968 heartbeater.cc:511] Master 127.1.8.190:41873 requested a full tablet report, sending...
I20260924 17:47:43.937974 20779 ts_manager.cc:194] Registered new tserver with Master: ce1d7a67de754376b1b67083651ce231 (127.1.8.129:38187)
I20260924 17:47:43.938496 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.130:0
--local_ip_for_outbound_sockets=127.1.8.130
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41873
--builtin_ntp_servers=127.1.8.148:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:47:43.938982 20779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.129:44623
W20260924 17:47:44.022050 20971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:44.022339 20971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:44.022377 20971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:44.023666 20971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:44.023747 20971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:44.023792 20971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.130
I20260924 17:47:44.025198 20971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.8.130
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.130
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:44.025501 20971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:44.025818 20971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:44.031126 20977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:44.031299 20971 server_base.cc:1034] running on GCE node
W20260924 17:47:44.031107 20979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:44.031103 20976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:44.031642 20971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:44.031906 20971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:44.033097 20971 hybrid_clock.cc:648] HybridClock initialized: now 1790272064033077 us; error 33 us; skew 500 ppm
I20260924 17:47:44.034446 20971 webserver.cc:460] Webserver started at http://127.1.8.130:37575/ using document root <none> and password file <none>
I20260924 17:47:44.034687 20971 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:44.034761 20971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:44.034902 20971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:44.035997 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data/instance:
uuid: "5f7e4ac89e22460db539aec19d376b52"
format_stamp: "Formatted at 2026-09-24 17:47:44 on dist-test-slave-r7hf"
server_key: "34232b1e0492001329992c2ba46c288d"
server_key_iv: "13e84ea9747cd7563d9316bd4bdca0b8"
server_key_version: "encryptionkey@0"
I20260924 17:47:44.036376 20971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance:
uuid: "5f7e4ac89e22460db539aec19d376b52"
format_stamp: "Formatted at 2026-09-24 17:47:44 on dist-test-slave-r7hf"
server_key: "34232b1e0492001329992c2ba46c288d"
server_key_iv: "13e84ea9747cd7563d9316bd4bdca0b8"
server_key_version: "encryptionkey@0"
I20260924 17:47:44.037791 20971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:44.038646 20985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:44.038828 20971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:44.038929 20971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal
uuid: "5f7e4ac89e22460db539aec19d376b52"
format_stamp: "Formatted at 2026-09-24 17:47:44 on dist-test-slave-r7hf"
server_key: "34232b1e0492001329992c2ba46c288d"
server_key_iv: "13e84ea9747cd7563d9316bd4bdca0b8"
server_key_version: "encryptionkey@0"
I20260924 17:47:44.039016 20971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:44.051119 20971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:44.051466 20971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:44.051632 20971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:44.051877 20971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:44.052258 20971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:44.052316 20971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:44.052362 20971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:44.052387 20971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:44.058543 20971 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.130:38537
I20260924 17:47:44.058588 21098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.130:38537 every 8 connection(s)
I20260924 17:47:44.058936 20971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/data/info.pb
I20260924 17:47:44.063956 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 20971
I20260924 17:47:44.064064 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal/instance
I20260924 17:47:44.064265 1058 external_mini_cluster.cc:1428] Setting key 1e0901342eb82a3903b306018e4602a7
I20260924 17:47:44.064491 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.059675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:56009 (local address 127.1.8.190:41873)
0924 17:47:44.059751 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.059753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.059998 (+ 245us) server_negotiation.cc:407] Connection header received
0924 17:47:44.060559 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.060560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.060572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.060629 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:44.061334 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.061853 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.062938 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.063133 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.063221 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.064141 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:44.064144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:44.064144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:44.064150 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:44.064168 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:44.064172 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:44.064229 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.064360 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.064397 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:47:44.064693 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.059666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41873 (local address 127.1.8.130:56009)
0924 17:47:44.059868 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.059904 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.060446 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.060723 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.060727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.060735 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:44.061230 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.061235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.061957 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.061958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.062835 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.062837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.063220 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.063221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.063363 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.063895 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:44.063895 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:44.063940 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:44.063970 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:44.064253 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:44.064259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:44.064262 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.064346 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.064466 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260924 17:47:44.065176 21099 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41873
I20260924 17:47:44.065267 21099 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:44.065452 21099 heartbeater.cc:511] Master 127.1.8.190:41873 requested a full tablet report, sending...
I20260924 17:47:44.065476 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.8.131:0
--local_ip_for_outbound_sockets=127.1.8.131
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41873
--builtin_ntp_servers=127.1.8.148:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 17:47:44.065979 20779 ts_manager.cc:194] Registered new tserver with Master: 5f7e4ac89e22460db539aec19d376b52 (127.1.8.130:38537)
I20260924 17:47:44.066362 20779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.130:56009
W20260924 17:47:44.149335 21102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:44.149644 21102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:44.149684 21102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:44.150972 21102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:44.151044 21102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:44.151089 21102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.8.131
I20260924 17:47:44.152880 21102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.8.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.8.131
--webserver_port=0
--tserver_master_addrs=127.1.8.190:41873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.8.131
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:44.153203 21102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:44.153542 21102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:44.158877 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:44.158869 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:44.158869 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:44.159320 21102 server_base.cc:1034] running on GCE node
I20260924 17:47:44.159571 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:44.159839 21102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:44.161020 21102 hybrid_clock.cc:648] HybridClock initialized: now 1790272064160996 us; error 33 us; skew 500 ppm
I20260924 17:47:44.162503 21102 webserver.cc:460] Webserver started at http://127.1.8.131:40071/ using document root <none> and password file <none>
I20260924 17:47:44.162766 21102 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:44.162837 21102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:44.162984 21102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:44.164103 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data/instance:
uuid: "d87db86dff644c55973bae86955427b1"
format_stamp: "Formatted at 2026-09-24 17:47:44 on dist-test-slave-r7hf"
server_key: "9f365bb7e4c609d869866c48d88ffee0"
server_key_iv: "fc1bb824337e22af1b009e248ef5ef0c"
server_key_version: "encryptionkey@0"
I20260924 17:47:44.164484 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance:
uuid: "d87db86dff644c55973bae86955427b1"
format_stamp: "Formatted at 2026-09-24 17:47:44 on dist-test-slave-r7hf"
server_key: "9f365bb7e4c609d869866c48d88ffee0"
server_key_iv: "fc1bb824337e22af1b009e248ef5ef0c"
server_key_version: "encryptionkey@0"
I20260924 17:47:44.165827 21102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:47:44.166904 21116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:44.167068 21102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:44.167179 21102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal
uuid: "d87db86dff644c55973bae86955427b1"
format_stamp: "Formatted at 2026-09-24 17:47:44 on dist-test-slave-r7hf"
server_key: "9f365bb7e4c609d869866c48d88ffee0"
server_key_iv: "fc1bb824337e22af1b009e248ef5ef0c"
server_key_version: "encryptionkey@0"
I20260924 17:47:44.167272 21102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:44.191432 21102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:44.191730 21102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:44.191836 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:44.192022 21102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 17:47:44.192373 21102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 17:47:44.192410 21102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:44.192469 21102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 17:47:44.192497 21102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:44.199236 21102 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.131:42713
I20260924 17:47:44.199275 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.131:42713 every 8 connection(s)
I20260924 17:47:44.199688 21102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/data/info.pb
I20260924 17:47:44.200723 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21102
I20260924 17:47:44.200840 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal/instance
I20260924 17:47:44.201057 1058 external_mini_cluster.cc:1428] Setting key b51c719dceec23f243ac4662f2a5d4ca
I20260924 17:47:44.206070 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.200521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.131:51905 (local address 127.1.8.190:41873)
0924 17:47:44.202541 (+ 2020us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.202543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.202551 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:47:44.202571 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.202572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.202581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.202628 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:44.203382 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.203863 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.204828 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.205006 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.205058 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.205864 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:44.205865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:44.205865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:44.205870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:44.205882 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:44.205886 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:44.205927 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.205988 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.206009 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986}
I20260924 17:47:44.206341 21232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.200504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:41873 (local address 127.1.8.131:51905)
0924 17:47:44.200762 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.200834 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.201456 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.202740 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.202744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.202771 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:47:44.203299 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.203304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.203941 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.203942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.204760 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.204762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.205080 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.205081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.205198 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.205721 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 17:47:44.205721 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:47:44.205761 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 17:47:44.205786 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:47:44.205929 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:47:44.205932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:47:44.205934 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.205983 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.206065 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260924 17:47:44.206866 21230 heartbeater.cc:347] Connected to a master server at 127.1.8.190:41873
I20260924 17:47:44.206954 21230 heartbeater.cc:464] Registering TS with master...
I20260924 17:47:44.207180 21230 heartbeater.cc:511] Master 127.1.8.190:41873 requested a full tablet report, sending...
I20260924 17:47:44.207605 20779 ts_manager.cc:194] Registered new tserver with Master: d87db86dff644c55973bae86955427b1 (127.1.8.131:42713)
I20260924 17:47:44.207993 20779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.8.131:51905
I20260924 17:47:44.212846 1058 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 17:47:44.217154 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.214196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.1.8.190:41873)
0924 17:47:44.214389 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.214467 (+ 78us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.214479 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:47:44.214504 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.214505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.214517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.214586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:44.214780 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.215265 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.215984 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.216161 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.216212 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.216875 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:44.216876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:44.216877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:44.216880 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:44.216891 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:44.216894 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:44.216932 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.216999 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.217018 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 17:47:44.219128 20779 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 17:47:44.220711 20779 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 17:47:44.227921 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.223532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.1.8.131:42713)
0924 17:47:44.223983 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.223988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.223997 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:44.224035 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.224038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.224053 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.224228 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:44.224364 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.225417 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.226873 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.227476 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.227577 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.227668 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.227783 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.227823 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":74,"threads_started":1}
I20260924 17:47:44.227921 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.223427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:42713 (local address 127.0.0.1:45654)
0924 17:47:44.223853 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.223864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.223908 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.224224 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.224225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.224228 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:44.224276 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.224278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.225578 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.225579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.226784 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.226786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.227585 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.227586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.227681 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.227696 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.227770 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.227827 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":88,"threads_started":1}
I20260924 17:47:44.227978 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.223335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.1.8.129:38187)
0924 17:47:44.223566 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.223572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.223582 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:47:44.223635 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.223637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.223659 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.223749 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:44.223991 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.225133 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.226644 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.227418 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.227499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.227604 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.227838 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.227884 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":114,"threads_started":1}
I20260924 17:47:44.228097 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.223736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.1.8.130:38537)
0924 17:47:44.224111 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.224115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.224382 (+ 267us) server_negotiation.cc:407] Connection header received
0924 17:47:44.224458 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.224460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.224474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.224569 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:44.224677 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.225360 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.226776 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.227463 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.227521 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.227604 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.227963 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.228010 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":116,"threads_started":1}
I20260924 17:47:44.228228 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.223173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:38187 (local address 127.0.0.1:42960)
0924 17:47:44.223355 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.223404 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.223490 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.223798 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.223802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.223804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:44.223885 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.223888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.225234 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.225235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.226551 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.226554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.227645 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.227646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.227724 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.227728 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.228089 (+ 361us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.228161 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260924 17:47:44.227924 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.223611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.130:38537 (local address 127.0.0.1:47868)
0924 17:47:44.224323 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.224333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.224383 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.224567 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.224568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.224569 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:44.224610 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.224612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.225454 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.225455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.226672 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.226675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.227643 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.227645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.227729 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.227732 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.227780 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.227834 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":29,"thread_start_us":56,"threads_started":1}
I20260924 17:47:44.229313 20902 tablet_service.cc:1467] Processing CreateTablet for tablet f10f6abfc90b499bb17127e378dc500f (DEFAULT_TABLE table=test-table [id=e2d7bf71d6d34d00a91429cd87536211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:44.229375 21164 tablet_service.cc:1467] Processing CreateTablet for tablet f10f6abfc90b499bb17127e378dc500f (DEFAULT_TABLE table=test-table [id=e2d7bf71d6d34d00a91429cd87536211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:44.229681 20902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10f6abfc90b499bb17127e378dc500f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:44.229693 21164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10f6abfc90b499bb17127e378dc500f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:44.230909 21033 tablet_service.cc:1467] Processing CreateTablet for tablet f10f6abfc90b499bb17127e378dc500f (DEFAULT_TABLE table=test-table [id=e2d7bf71d6d34d00a91429cd87536211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 17:47:44.231293 21033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10f6abfc90b499bb17127e378dc500f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:44.232751 21244 tablet_bootstrap.cc:492] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Bootstrap starting.
I20260924 17:47:44.233078 21245 tablet_bootstrap.cc:492] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: Bootstrap starting.
I20260924 17:47:44.233762 21244 tablet_bootstrap.cc:654] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:44.233762 21245 tablet_bootstrap.cc:654] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:44.234099 21245 log.cc:824] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:44.234112 21244 log.cc:824] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:44.234769 21246 tablet_bootstrap.cc:492] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Bootstrap starting.
I20260924 17:47:44.235019 21244 tablet_bootstrap.cc:492] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: No bootstrap required, opened a new log
I20260924 17:47:44.235103 21244 ts_tablet_manager.cc:1397] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:47:44.235361 21246 tablet_bootstrap.cc:654] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:44.235656 21246 log.cc:824] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:44.236328 21246 tablet_bootstrap.cc:492] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: No bootstrap required, opened a new log
I20260924 17:47:44.236413 21246 ts_tablet_manager.cc:1397] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 17:47:44.236539 21245 tablet_bootstrap.cc:492] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: No bootstrap required, opened a new log
I20260924 17:47:44.236617 21245 ts_tablet_manager.cc:1397] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 17:47:44.238058 21244 raft_consensus.cc:348] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.238245 21244 raft_consensus.cc:374] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:44.238282 21244 raft_consensus.cc:729] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d87db86dff644c55973bae86955427b1, State: Initialized, Role: FOLLOWER
I20260924 17:47:44.238204 21246 raft_consensus.cc:348] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.238430 21244 consensus_queue.cc:260] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.238628 21246 raft_consensus.cc:374] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:44.238703 21246 raft_consensus.cc:729] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7e4ac89e22460db539aec19d376b52, State: Initialized, Role: FOLLOWER
I20260924 17:47:44.238721 21244 ts_tablet_manager.cc:1428] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 17:47:44.238848 21230 heartbeater.cc:503] Master 127.1.8.190:41873 was elected leader, sending a full tablet report...
I20260924 17:47:44.238876 21246 consensus_queue.cc:260] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.239141 21246 ts_tablet_manager.cc:1428] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 17:47:44.239311 21099 heartbeater.cc:503] Master 127.1.8.190:41873 was elected leader, sending a full tablet report...
I20260924 17:47:44.239409 21245 raft_consensus.cc:348] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.239607 21245 raft_consensus.cc:374] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:44.239652 21245 raft_consensus.cc:729] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1d7a67de754376b1b67083651ce231, State: Initialized, Role: FOLLOWER
I20260924 17:47:44.239765 21245 consensus_queue.cc:260] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.240036 21245 ts_tablet_manager.cc:1428] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 17:47:44.240098 20968 heartbeater.cc:503] Master 127.1.8.190:41873 was elected leader, sending a full tablet report...
W20260924 17:47:44.309470 21100 tablet.cc:2369] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 17:47:44.314469 21251 raft_consensus.cc:482] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 17:47:44.314634 21251 raft_consensus.cc:504] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.315001 21251 leader_election.cc:288] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1d7a67de754376b1b67083651ce231 (127.1.8.129:38187), d87db86dff644c55973bae86955427b1 (127.1.8.131:42713)
I20260924 17:47:44.319036 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.315503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:34131 (local address 127.1.8.131:42713)
0924 17:47:44.315550 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.315552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.315561 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:44.315655 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.315656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.315674 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.315744 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:44.315891 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.316562 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.318005 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.318611 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.318688 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.318699 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.318921 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.318962 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 17:47:44.319036 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.315300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.8.130:34143 (local address 127.1.8.129:38187)
0924 17:47:44.315357 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.315359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.315377 (+ 18us) server_negotiation.cc:407] Connection header received
0924 17:47:44.315404 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.315405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.315414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.315501 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 17:47:44.315663 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.316480 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.318006 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.318605 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.318689 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.318699 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.318921 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.318962 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 17:47:44.319054 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.315217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:38187 (local address 127.1.8.130:34143)
0924 17:47:44.315273 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.315286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.315333 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.315508 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.315509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.315510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:44.315586 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.315588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.316572 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.316573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.317905 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.317907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.318736 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.318738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.318824 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.318829 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.318902 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.318952 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":62}
I20260924 17:47:44.319059 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.315344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.131:42713 (local address 127.1.8.130:34131)
0924 17:47:44.315510 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.315520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:47:44.315563 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:47:44.315769 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:47:44.315770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:47:44.315771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 17:47:44.315820 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.315821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.316630 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.316630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.317900 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:47:44.317904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.318737 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.318738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:47:44.318829 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.318831 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 17:47:44.318902 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 17:47:44.318950 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":13,"thread_start_us":104,"threads_started":1}
I20260924 17:47:44.319500 20922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10f6abfc90b499bb17127e378dc500f" candidate_uuid: "5f7e4ac89e22460db539aec19d376b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1d7a67de754376b1b67083651ce231" is_pre_election: true
I20260924 17:47:44.319557 21184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10f6abfc90b499bb17127e378dc500f" candidate_uuid: "5f7e4ac89e22460db539aec19d376b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87db86dff644c55973bae86955427b1" is_pre_election: true
I20260924 17:47:44.319681 20922 raft_consensus.cc:2454] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f7e4ac89e22460db539aec19d376b52 in term 0.
I20260924 17:47:44.319731 21184 raft_consensus.cc:2454] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f7e4ac89e22460db539aec19d376b52 in term 0.
I20260924 17:47:44.319988 20987 leader_election.cc:302] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7e4ac89e22460db539aec19d376b52, d87db86dff644c55973bae86955427b1; no voters:
I20260924 17:47:44.320192 21251 raft_consensus.cc:2781] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 17:47:44.320278 21251 raft_consensus.cc:482] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 17:47:44.320303 21251 raft_consensus.cc:3037] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:44.321125 21251 raft_consensus.cc:504] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.321244 21251 leader_election.cc:288] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [CANDIDATE]: Term 1 election: Requested vote from peers ce1d7a67de754376b1b67083651ce231 (127.1.8.129:38187), d87db86dff644c55973bae86955427b1 (127.1.8.131:42713)
I20260924 17:47:44.321533 20922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10f6abfc90b499bb17127e378dc500f" candidate_uuid: "5f7e4ac89e22460db539aec19d376b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1d7a67de754376b1b67083651ce231"
I20260924 17:47:44.321619 21184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10f6abfc90b499bb17127e378dc500f" candidate_uuid: "5f7e4ac89e22460db539aec19d376b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d87db86dff644c55973bae86955427b1"
I20260924 17:47:44.321616 20922 raft_consensus.cc:3037] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:44.321730 21184 raft_consensus.cc:3037] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:44.322492 20922 raft_consensus.cc:2454] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f7e4ac89e22460db539aec19d376b52 in term 1.
I20260924 17:47:44.322613 21184 raft_consensus.cc:2454] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f7e4ac89e22460db539aec19d376b52 in term 1.
I20260924 17:47:44.322671 20987 leader_election.cc:302] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f7e4ac89e22460db539aec19d376b52, ce1d7a67de754376b1b67083651ce231; no voters:
I20260924 17:47:44.322750 21251 raft_consensus.cc:2781] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:44.322926 21251 raft_consensus.cc:686] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7e4ac89e22460db539aec19d376b52, State: Running, Role: LEADER
I20260924 17:47:44.323035 21251 consensus_queue.cc:237] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } }
I20260924 17:47:44.323904 20779 catalog_manager.cc:5495] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f7e4ac89e22460db539aec19d376b52 (127.1.8.130). New cstate: current_term: 1 leader_uuid: "5f7e4ac89e22460db539aec19d376b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f7e4ac89e22460db539aec19d376b52" member_type: VOTER last_known_addr { host: "127.1.8.130" port: 38537 } health_report { overall_health: HEALTHY } } }
I20260924 17:47:44.347273 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.345067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.1.8.130:38537)
0924 17:47:44.345111 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:44.345113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:44.345123 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:47:44.345172 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:44.345173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:44.345191 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:44.345262 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:47:44.345368 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.345830 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.346620 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:44.346805 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:44.346861 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:44.346980 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:47:44.347079 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:47:44.347136 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:44.347164 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 17:47:44.347178 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:47:44.349218 20922 raft_consensus.cc:1261] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 1 FOLLOWER]: Refusing update from remote peer 5f7e4ac89e22460db539aec19d376b52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:44.349239 21184 raft_consensus.cc:1261] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 1 FOLLOWER]: Refusing update from remote peer 5f7e4ac89e22460db539aec19d376b52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:47:44.349670 21251 consensus_queue.cc:1035] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d87db86dff644c55973bae86955427b1" member_type: VOTER last_known_addr { host: "127.1.8.131" port: 42713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:44.349833 21254 consensus_queue.cc:1035] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1d7a67de754376b1b67083651ce231" member_type: VOTER last_known_addr { host: "127.1.8.129" port: 38187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:47:44.352598 21259 mvcc.cc:205] Tried to move back new op lower bound from 7332954375571185664 to 7332954375467737088. Current Snapshot: MvccSnapshot[applied={T|T < 7332954375571185664}]
I20260924 17:47:44.354773 20780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:44.354864 20780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 17:47:44.355734 20780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc9c5252fc4f4d4eb3399385b145cb75: Sending DeleteTablet for 3 replicas of tablet f10f6abfc90b499bb17127e378dc500f
I20260924 17:47:44.356093 21164 tablet_service.cc:1514] Processing DeleteTablet for tablet f10f6abfc90b499bb17127e378dc500f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:44 UTC) from {username='slave'} at 127.0.0.1:45654
I20260924 17:47:44.356194 20902 tablet_service.cc:1514] Processing DeleteTablet for tablet f10f6abfc90b499bb17127e378dc500f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:44 UTC) from {username='slave'} at 127.0.0.1:42960
I20260924 17:47:44.356395 21268 tablet_replica.cc:330] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: stopping tablet replica
I20260924 17:47:44.356390 21033 tablet_service.cc:1514] Processing DeleteTablet for tablet f10f6abfc90b499bb17127e378dc500f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 17:47:44 UTC) from {username='slave'} at 127.0.0.1:47868
I20260924 17:47:44.356506 21268 raft_consensus.cc:2229] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:44.356629 21268 raft_consensus.cc:2258] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:44.356626 21269 tablet_replica.cc:330] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: stopping tablet replica
I20260924 17:47:44.356818 21267 tablet_replica.cc:330] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: stopping tablet replica
I20260924 17:47:44.356922 21267 raft_consensus.cc:2229] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 17:47:44.356941 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20840
I20260924 17:47:44.357019 21267 raft_consensus.cc:2258] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:44.357096 21268 ts_tablet_manager.cc:1905] T f10f6abfc90b499bb17127e378dc500f P ce1d7a67de754376b1b67083651ce231: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:44.357455 21267 ts_tablet_manager.cc:1905] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:44.359184 21267 ts_tablet_manager.cc:1918] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:44.359251 21267 log.cc:1196] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-2/wal/wals/f10f6abfc90b499bb17127e378dc500f
I20260924 17:47:44.359506 21267 ts_tablet_manager.cc:1939] T f10f6abfc90b499bb17127e378dc500f P d87db86dff644c55973bae86955427b1: Deleting consensus metadata
I20260924 17:47:44.359884 21269 raft_consensus.cc:2229] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 1 LEADER]: Raft consensus shutting down.
I20260924 17:47:44.360076 20765 catalog_manager.cc:4841] TS d87db86dff644c55973bae86955427b1 (127.1.8.131:42713): tablet f10f6abfc90b499bb17127e378dc500f (table test-table [id=e2d7bf71d6d34d00a91429cd87536211]) successfully deleted
I20260924 17:47:44.360193 21269 raft_consensus.cc:2258] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 17:47:44.360603 21269 ts_tablet_manager.cc:1905] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 17:47:44.361975 21269 ts_tablet_manager.cc:1918] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 17:47:44.362069 21269 log.cc:1196] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Deleting WAL directory at /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790271791394214-1058-0/minicluster-data/ts-1/wal/wals/f10f6abfc90b499bb17127e378dc500f
I20260924 17:47:44.362320 21269 ts_tablet_manager.cc:1939] T f10f6abfc90b499bb17127e378dc500f P 5f7e4ac89e22460db539aec19d376b52: Deleting consensus metadata
I20260924 17:47:44.362818 20765 catalog_manager.cc:4841] TS 5f7e4ac89e22460db539aec19d376b52 (127.1.8.130:38537): tablet f10f6abfc90b499bb17127e378dc500f (table test-table [id=e2d7bf71d6d34d00a91429cd87536211]) successfully deleted
W20260924 17:47:44.364890 20765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.8.129:38187 (error 108)
I20260924 17:47:44.365451 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:44.365301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.129:38187 (local address 127.0.0.1:42976)
0924 17:47:44.365358 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 17:47:44.365419 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.129:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:47:44.365468 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20971
W20260924 17:47:44.366443 20765 catalog_manager.cc:4584] TS ce1d7a67de754376b1b67083651ce231 (127.1.8.129:38187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f10f6abfc90b499bb17127e378dc500f: Network error: Client connection negotiation failed: client connection to 127.1.8.129:38187: connect: Connection refused (error 111)
I20260924 17:47:44.370603 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21102
I20260924 17:47:44.375622 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 20749
2026-09-24T17:47:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (764 ms)
[----------] 6 tests from EncryptionPolicyTest (8813 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:44 dist-test-slave-r7hf krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:44 dist-test-slave-r7hf krb5kdc[21273](info): set up 2 sockets
Sep 24 17:47:44 dist-test-slave-r7hf krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260924 17:47:45.410382 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:45 dist-test-slave-r7hf krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:45.533423 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:45.615341 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:45.615614 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:45.615656 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:45.616858 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:45.616943 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:45.616966 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:45.616987 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:45.617017 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:45.618570 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:45.618877 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:45.619117 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:47:45.624514 21286 server_base.cc:1034] running on GCE node
W20260924 17:47:45.624476 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:45.624476 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:45.624507 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:45.625139 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:45.625415 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:45.626559 21286 hybrid_clock.cc:648] HybridClock initialized: now 1790272065626543 us; error 28 us; skew 500 ppm
Sep 24 17:47:45 dist-test-slave-r7hf krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:45.629036 21286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:45.629444 21286 webserver.cc:460] Webserver started at http://127.0.0.1:42875/ using document root <none> and password file <none>
I20260924 17:47:45.629644 21286 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:45.629686 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:45.629786 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:45.630895 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "d4a0098d055246e8bd81ab10fd279ec8"
format_stamp: "Formatted at 2026-09-24 17:47:45 on dist-test-slave-r7hf"
server_key: "6f88b81b46492493c68b5a47099e4625"
server_key_iv: "40bf7a31c806875c5d9f368a9d18be24"
server_key_version: "encryptionkey@0"
I20260924 17:47:45.631206 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "d4a0098d055246e8bd81ab10fd279ec8"
format_stamp: "Formatted at 2026-09-24 17:47:45 on dist-test-slave-r7hf"
server_key: "6f88b81b46492493c68b5a47099e4625"
server_key_iv: "40bf7a31c806875c5d9f368a9d18be24"
server_key_version: "encryptionkey@0"
I20260924 17:47:45.632498 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:47:45.633260 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:45.633522 21286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:45.633611 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "d4a0098d055246e8bd81ab10fd279ec8"
format_stamp: "Formatted at 2026-09-24 17:47:45 on dist-test-slave-r7hf"
server_key: "6f88b81b46492493c68b5a47099e4625"
server_key_iv: "40bf7a31c806875c5d9f368a9d18be24"
server_key_version: "encryptionkey@0"
I20260924 17:47:45.633673 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:45.665840 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:45.668048 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:45.668169 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:45.672503 21286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43059
I20260924 17:47:45.672540 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43059 every 8 connection(s)
I20260924 17:47:45.672962 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:45.674034 21354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:45.676394 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8: Bootstrap starting.
I20260924 17:47:45.677014 21354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:45.677309 21354 log.cc:824] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:45.678018 21354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8: No bootstrap required, opened a new log
I20260924 17:47:45.678609 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21286
I20260924 17:47:45.678689 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:45.678901 1058 external_mini_cluster.cc:1428] Setting key 45a292316c630eb9eca1706d23b46c0f
I20260924 17:47:45.680454 21354 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0098d055246e8bd81ab10fd279ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260924 17:47:45.680672 21354 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:45.680717 21354 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a0098d055246e8bd81ab10fd279ec8, State: Initialized, Role: FOLLOWER
I20260924 17:47:45.680842 21354 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0098d055246e8bd81ab10fd279ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260924 17:47:45.680929 21354 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:45.680963 21354 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:45.681012 21354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:45.681607 21354 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0098d055246e8bd81ab10fd279ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260924 17:47:45.681718 21354 leader_election.cc:302] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a0098d055246e8bd81ab10fd279ec8; no voters:
I20260924 17:47:45.681934 21354 leader_election.cc:288] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:45.681942 21359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:45.682129 21359 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [term 1 LEADER]: Becoming Leader. State: Replica: d4a0098d055246e8bd81ab10fd279ec8, State: Running, Role: LEADER
I20260924 17:47:45.682222 21354 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:45.682226 21359 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0098d055246e8bd81ab10fd279ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } }
I20260924 17:47:45.682626 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a0098d055246e8bd81ab10fd279ec8. Latest consensus state: current_term: 1 leader_uuid: "d4a0098d055246e8bd81ab10fd279ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0098d055246e8bd81ab10fd279ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } } }
I20260924 17:47:45.682691 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:45.682881 21364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:45.682837 21361 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a0098d055246e8bd81ab10fd279ec8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a0098d055246e8bd81ab10fd279ec8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43059 } } }
I20260924 17:47:45.682936 21361 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:45.683410 21364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:45.685982 21364 catalog_manager.cc:1347] Generated new cluster ID: 32dede4173f54db5874f9a669e2f0b81
I20260924 17:47:45.686036 21364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:47:45 dist-test-slave-r7hf krb5kdc[21273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:47:45.693081 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:45.679531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33723 (local address 127.0.0.1:43059)
0924 17:47:45.679724 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:45.679734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:45.679773 (+ 39us) server_negotiation.cc:407] Connection header received
0924 17:47:45.680673 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:45.680683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:45.680750 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:45.680903 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:45.681640 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:45.684166 (+ 2526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:45.684971 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:45.685567 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:45.685677 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:45.688616 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:45.688621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:45.688622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:45.688635 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:45.691281 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:45.691781 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:45.691785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:45.691785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:45.691901 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:45.692152 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:45.692153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:45.692153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:45.692464 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:45.692614 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:45.692760 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 17:47:45.692860 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260924 17:47:45.694218 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:45.706815 21358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:45.695844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55717 (local address 127.0.0.1:43059)
0924 17:47:45.695879 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:45.695881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:45.695890 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:47:45.695916 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:45.695917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:45.695926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:45.695991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:45.696571 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:45.697394 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:45.698386 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:45.698744 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:45.698817 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:45.703824 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:45.703826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:45.703826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:45.703835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:45.705459 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:45.705924 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:45.705925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:45.705926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:45.705983 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:45.706223 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:45.706224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:45.706224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:45.706357 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:45.706473 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:45.706654 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 17:47:45.706715 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:47:45.720957 21364 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:45.721590 21364 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:45.725955 21364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a0098d055246e8bd81ab10fd279ec8: Generated new TSK 0
I20260924 17:47:45.726161 21364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:45.732101 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21286
W20260924 17:47:45.736578 21271 connection.cc:535] client connection to 127.0.0.1:43059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43059): Connection reset by peer (error 104)
2026-09-24T17:47:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:45 dist-test-slave-r7hf krb5kdc[21385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:45 dist-test-slave-r7hf krb5kdc[21385](info): set up 2 sockets
Sep 24 17:47:45 dist-test-slave-r7hf krb5kdc[21385](info): commencing operation
krb5kdc: starting...
W20260924 17:47:46.810542 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:46 dist-test-slave-r7hf krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:46.933631 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:47.016680 21398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:47.016963 21398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:47.017005 21398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:47.018211 21398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:47.018289 21398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:47.018313 21398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:47.018335 21398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:47.018365 21398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:47.019891 21398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:47.020182 21398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:47.020419 21398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:47.025457 21406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:47.025552 21398 server_base.cc:1034] running on GCE node
W20260924 17:47:47.025470 21403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:47.025460 21404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:47.025915 21398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:47.026255 21398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:47.027398 21398 hybrid_clock.cc:648] HybridClock initialized: now 1790272067027376 us; error 49 us; skew 500 ppm
Sep 24 17:47:47 dist-test-slave-r7hf krb5kdc[21385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:47.030120 21398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:47.030627 21398 webserver.cc:460] Webserver started at http://127.0.0.1:36933/ using document root <none> and password file <none>
I20260924 17:47:47.030851 21398 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:47.030896 21398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:47.030989 21398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:47.032119 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "e778b30e8bf042b4aa2f537775b2157a"
format_stamp: "Formatted at 2026-09-24 17:47:47 on dist-test-slave-r7hf"
server_key: "515c97432fb6a0cbb013cf6e4472e489"
server_key_iv: "17cdcda47757fbf5ab4e123d7eac093e"
server_key_version: "encryptionkey@0"
I20260924 17:47:47.032444 21398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "e778b30e8bf042b4aa2f537775b2157a"
format_stamp: "Formatted at 2026-09-24 17:47:47 on dist-test-slave-r7hf"
server_key: "515c97432fb6a0cbb013cf6e4472e489"
server_key_iv: "17cdcda47757fbf5ab4e123d7eac093e"
server_key_version: "encryptionkey@0"
I20260924 17:47:47.033754 21398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:47.034508 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:47.034714 21398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:47.034788 21398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "e778b30e8bf042b4aa2f537775b2157a"
format_stamp: "Formatted at 2026-09-24 17:47:47 on dist-test-slave-r7hf"
server_key: "515c97432fb6a0cbb013cf6e4472e489"
server_key_iv: "17cdcda47757fbf5ab4e123d7eac093e"
server_key_version: "encryptionkey@0"
I20260924 17:47:47.034847 21398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:47.045162 21398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:47.047173 21398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:47.047298 21398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:47.051461 21398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34485
I20260924 17:47:47.051515 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34485 every 8 connection(s)
I20260924 17:47:47.051889 21398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:47.052834 21466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:47.055173 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a: Bootstrap starting.
I20260924 17:47:47.055765 21466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:47.056047 21466 log.cc:824] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:47.056761 21466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a: No bootstrap required, opened a new log
I20260924 17:47:47.058331 21466 raft_consensus.cc:348] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e778b30e8bf042b4aa2f537775b2157a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34485 } }
I20260924 17:47:47.058564 21466 raft_consensus.cc:374] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:47.058619 21466 raft_consensus.cc:729] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e778b30e8bf042b4aa2f537775b2157a, State: Initialized, Role: FOLLOWER
I20260924 17:47:47.058636 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21398
I20260924 17:47:47.058744 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:47.058828 21466 consensus_queue.cc:260] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e778b30e8bf042b4aa2f537775b2157a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34485 } }
I20260924 17:47:47.058933 21466 raft_consensus.cc:388] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:47.058964 1058 external_mini_cluster.cc:1428] Setting key 7b76bd69059c8ae19a39e5446e58cea3
I20260924 17:47:47.058980 21466 raft_consensus.cc:482] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:47.059029 21466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:47.060201 21466 raft_consensus.cc:504] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e778b30e8bf042b4aa2f537775b2157a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34485 } }
I20260924 17:47:47.060319 21466 leader_election.cc:302] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e778b30e8bf042b4aa2f537775b2157a; no voters:
I20260924 17:47:47.060480 21466 leader_election.cc:288] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:47.060540 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:47.060716 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [term 1 LEADER]: Becoming Leader. State: Replica: e778b30e8bf042b4aa2f537775b2157a, State: Running, Role: LEADER
I20260924 17:47:47.060777 21466 sys_catalog.cc:564] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:47.060849 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e778b30e8bf042b4aa2f537775b2157a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34485 } }
I20260924 17:47:47.061312 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e778b30e8bf042b4aa2f537775b2157a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e778b30e8bf042b4aa2f537775b2157a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34485 } } }
I20260924 17:47:47.061407 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:47.061610 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e778b30e8bf042b4aa2f537775b2157a. Latest consensus state: current_term: 1 leader_uuid: "e778b30e8bf042b4aa2f537775b2157a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e778b30e8bf042b4aa2f537775b2157a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34485 } } }
I20260924 17:47:47.061715 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:47.061915 21480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:47.062500 21480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:47.064325 21480 catalog_manager.cc:1347] Generated new cluster ID: e4718e9124d94a2fb57faa92d9c437c5
I20260924 17:47:47.064380 21480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:47.065423 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:47.059479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50601 (local address 127.0.0.1:34485)
0924 17:47:47.059724 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:47.059736 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:47.059785 (+ 49us) server_negotiation.cc:407] Connection header received
0924 17:47:47.059931 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:47.059939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:47.059997 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:47.060149 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:47.060416 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:47.061673 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:47.063020 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:47.063647 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:47.063803 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:47.064685 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:47.064689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:47.064690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:47.064900 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:47.064936 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:47.064951 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:47.065032 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:47.065172 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 17:47:47.065200 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":111,"threads_started":1}
I20260924 17:47:47.066473 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:47.074352 21470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:47.068032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41051 (local address 127.0.0.1:34485)
0924 17:47:47.068068 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:47.068070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:47.070635 (+ 2565us) server_negotiation.cc:407] Connection header received
0924 17:47:47.070706 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:47.070707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:47.070722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:47.070783 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:47.070979 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:47.071745 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:47.072782 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:47.073166 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:47.073241 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:47.074046 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:47.074047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:47.074048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:47.074054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:47.074071 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:47.074075 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:47.074138 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:47.074244 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 17:47:47.074258 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:47:47.084851 21480 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:47.085531 21480 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:47.090157 21480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e778b30e8bf042b4aa2f537775b2157a: Generated new TSK 0
I20260924 17:47:47.090368 21480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:47.099139 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21398
W20260924 17:47:47.103319 21383 connection.cc:535] client connection to 127.0.0.1:34485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34485): Connection reset by peer (error 104)
2026-09-24T17:47:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:47 dist-test-slave-r7hf krb5kdc[21497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:47 dist-test-slave-r7hf krb5kdc[21497](info): set up 2 sockets
Sep 24 17:47:47 dist-test-slave-r7hf krb5kdc[21497](info): commencing operation
krb5kdc: starting...
W20260924 17:47:48.158607 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:48 dist-test-slave-r7hf krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:48.281409 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:48.371811 21510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:48.372030 21510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:48.372053 21510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:48.373251 21510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:48.373298 21510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:48.373313 21510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:48.373327 21510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:48.373385 21510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:48.374953 21510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:48.375216 21510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:48.375453 21510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:48.380858 21515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:48.380903 21510 server_base.cc:1034] running on GCE node
W20260924 17:47:48.380815 21516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:48.380808 21518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:48.381392 21510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:48.381705 21510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:48.382887 21510 hybrid_clock.cc:648] HybridClock initialized: now 1790272068382874 us; error 47 us; skew 500 ppm
Sep 24 17:47:48 dist-test-slave-r7hf krb5kdc[21497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:48.385486 21510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:48.385886 21510 webserver.cc:460] Webserver started at http://127.0.0.1:42319/ using document root <none> and password file <none>
I20260924 17:47:48.386075 21510 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:48.386117 21510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:48.386219 21510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:48.387326 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "75436ecc5ef64c5b9e36da1e7307e5f4"
format_stamp: "Formatted at 2026-09-24 17:47:48 on dist-test-slave-r7hf"
server_key: "7cdd691c643c9bb92ea8ba4fe2318538"
server_key_iv: "4b213621cc2ae2be0e3eaad78de121c9"
server_key_version: "encryptionkey@0"
I20260924 17:47:48.387627 21510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "75436ecc5ef64c5b9e36da1e7307e5f4"
format_stamp: "Formatted at 2026-09-24 17:47:48 on dist-test-slave-r7hf"
server_key: "7cdd691c643c9bb92ea8ba4fe2318538"
server_key_iv: "4b213621cc2ae2be0e3eaad78de121c9"
server_key_version: "encryptionkey@0"
I20260924 17:47:48.388904 21510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 17:47:48.389672 21525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:48.389843 21510 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:48.389937 21510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "75436ecc5ef64c5b9e36da1e7307e5f4"
format_stamp: "Formatted at 2026-09-24 17:47:48 on dist-test-slave-r7hf"
server_key: "7cdd691c643c9bb92ea8ba4fe2318538"
server_key_iv: "4b213621cc2ae2be0e3eaad78de121c9"
server_key_version: "encryptionkey@0"
I20260924 17:47:48.389998 21510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:48.393340 21510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:48.395381 21510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:48.395488 21510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:48.399341 21510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37877
I20260924 17:47:48.399408 21577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37877 every 8 connection(s)
I20260924 17:47:48.399744 21510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:48.400422 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21510
I20260924 17:47:48.400540 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:48.400774 1058 external_mini_cluster.cc:1428] Setting key 56f743364e16b19304829065c81baf12
I20260924 17:47:48.400916 21578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:48.403512 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:48.401348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45737 (local address 127.0.0.1:37877)
0924 17:47:48.401580 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:48.401590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:48.401621 (+ 31us) server_negotiation.cc:407] Connection header received
0924 17:47:48.402115 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:48.402124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:48.402168 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:48.402341 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:48.403062 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:48.403067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:48.403067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:48.403071 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:48.403115 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:48.403129 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:48.403213 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:48.403282 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 17:47:48.403323 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
I20260924 17:47:48.404487 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:48.404553 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4: Bootstrap starting.
I20260924 17:47:48.405158 21578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:48.405498 21578 log.cc:824] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:48.406387 21578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4: No bootstrap required, opened a new log
I20260924 17:47:48.406963 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:48.405935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41945 (local address 127.0.0.1:37877)
0924 17:47:48.405976 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:48.405977 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:48.406078 (+ 101us) server_negotiation.cc:407] Connection header received
0924 17:47:48.406108 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:48.406109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:48.406113 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:48.406183 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:48.406755 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:48.406756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:48.406756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:48.406760 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:48.406776 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:48.406780 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:48.406845 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:48.406879 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 17:47:48.406892 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:47:48.408448 21578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37877 } }
I20260924 17:47:48.408586 21578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:48.408608 21578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75436ecc5ef64c5b9e36da1e7307e5f4, State: Initialized, Role: FOLLOWER
I20260924 17:47:48.408751 21578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37877 } }
I20260924 17:47:48.408845 21578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:48.408903 21578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:48.408957 21578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:48.409490 21578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37877 } }
I20260924 17:47:48.409600 21578 leader_election.cc:302] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75436ecc5ef64c5b9e36da1e7307e5f4; no voters:
I20260924 17:47:48.409776 21578 leader_election.cc:288] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:48.409860 21588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:48.410014 21578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:48.410028 21588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 75436ecc5ef64c5b9e36da1e7307e5f4, State: Running, Role: LEADER
I20260924 17:47:48.410158 21588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37877 } }
I20260924 17:47:48.410574 21590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75436ecc5ef64c5b9e36da1e7307e5f4. Latest consensus state: current_term: 1 leader_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37877 } } }
I20260924 17:47:48.410696 21590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:48.410552 21589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75436ecc5ef64c5b9e36da1e7307e5f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37877 } } }
I20260924 17:47:48.410899 21597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:48.410912 21589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:48.411455 21597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:48.413059 21597 catalog_manager.cc:1347] Generated new cluster ID: 78ac87fd7a644abb934b2f2103c6be28
I20260924 17:47:48.413115 21597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:48.429797 21597 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:48.430004 21597 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:48.437827 21597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75436ecc5ef64c5b9e36da1e7307e5f4: Generated new TSK 0
I20260924 17:47:48.438053 21597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:48.442086 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21510
2026-09-24T17:47:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:48 dist-test-slave-r7hf krb5kdc[21609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:48 dist-test-slave-r7hf krb5kdc[21609](info): set up 2 sockets
Sep 24 17:47:48 dist-test-slave-r7hf krb5kdc[21609](info): commencing operation
krb5kdc: starting...
W20260924 17:47:49.495098 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:49 dist-test-slave-r7hf krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:49.622311 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:49.705818 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:49.706049 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:49.706073 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:49.707302 21622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:49.707350 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:49.707365 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:49.707378 21622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:49.707391 21622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:49.708841 21622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:49.709059 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:49.709246 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:49.714119 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:49.714114 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:49.714119 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:49.714506 21622 server_base.cc:1034] running on GCE node
I20260924 17:47:49.714792 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:49.715198 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:49.716395 21622 hybrid_clock.cc:648] HybridClock initialized: now 1790272069716366 us; error 39 us; skew 500 ppm
Sep 24 17:47:49 dist-test-slave-r7hf krb5kdc[21609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:49.719528 21622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:49.719957 21622 webserver.cc:460] Webserver started at http://127.0.0.1:34083/ using document root <none> and password file <none>
I20260924 17:47:49.720157 21622 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:49.720201 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:49.720290 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:49.721349 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "6a0385be959348458f6437b56754770f"
format_stamp: "Formatted at 2026-09-24 17:47:49 on dist-test-slave-r7hf"
server_key: "b0954c6b760834ccc2ddd1ef6c712285"
server_key_iv: "772c7e003798685b69dda468dfeb7069"
server_key_version: "encryptionkey@0"
I20260924 17:47:49.721695 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "6a0385be959348458f6437b56754770f"
format_stamp: "Formatted at 2026-09-24 17:47:49 on dist-test-slave-r7hf"
server_key: "b0954c6b760834ccc2ddd1ef6c712285"
server_key_iv: "772c7e003798685b69dda468dfeb7069"
server_key_version: "encryptionkey@0"
I20260924 17:47:49.723107 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:49.723850 21637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:49.724044 21622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:47:49.724134 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "6a0385be959348458f6437b56754770f"
format_stamp: "Formatted at 2026-09-24 17:47:49 on dist-test-slave-r7hf"
server_key: "b0954c6b760834ccc2ddd1ef6c712285"
server_key_iv: "772c7e003798685b69dda468dfeb7069"
server_key_version: "encryptionkey@0"
I20260924 17:47:49.724195 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:49.739847 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:49.741920 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:49.742040 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:49.746093 21622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33343
I20260924 17:47:49.746129 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33343 every 8 connection(s)
I20260924 17:47:49.746531 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:49.747507 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21622
I20260924 17:47:49.747597 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:49.747623 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:49.747843 1058 external_mini_cluster.cc:1428] Setting key 9abf66415c221ee6e8f7fbc5465b08af
I20260924 17:47:49.750622 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f: Bootstrap starting.
I20260924 17:47:49.751230 21690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:49.751516 21690 log.cc:824] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:49.752213 21690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f: No bootstrap required, opened a new log
I20260924 17:47:49.754359 21690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0385be959348458f6437b56754770f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260924 17:47:49.754572 21690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:49.754609 21690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0385be959348458f6437b56754770f, State: Initialized, Role: FOLLOWER
Sep 24 17:47:49 dist-test-slave-r7hf krb5kdc[21609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:47:49.754786 21690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0385be959348458f6437b56754770f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260924 17:47:49.754875 21690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:49.754912 21690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:49.754971 21690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:49.755630 21690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0385be959348458f6437b56754770f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260924 17:47:49.755753 21690 leader_election.cc:302] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a0385be959348458f6437b56754770f; no voters:
I20260924 17:47:49.755956 21690 leader_election.cc:288] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:49.755986 21695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:49.756099 21695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [term 1 LEADER]: Becoming Leader. State: Replica: 6a0385be959348458f6437b56754770f, State: Running, Role: LEADER
I20260924 17:47:49.756243 21690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:49.756229 21695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0385be959348458f6437b56754770f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260924 17:47:49.756587 21696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a0385be959348458f6437b56754770f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0385be959348458f6437b56754770f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } } }
I20260924 17:47:49.756680 21696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:49.756933 21697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a0385be959348458f6437b56754770f. Latest consensus state: current_term: 1 leader_uuid: "6a0385be959348458f6437b56754770f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0385be959348458f6437b56754770f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } } }
I20260924 17:47:49.756983 21702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:49.757614 21702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:49.757053 21697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:49.759734 21702 catalog_manager.cc:1347] Generated new cluster ID: 47650203d5494befb7f1585b0c02f0de
I20260924 17:47:49.759806 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:49.748441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41721 (local address 127.0.0.1:33343)
0924 17:47:49.748639 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:49.748647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:49.748674 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:47:49.749127 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:49.749133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:49.749183 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:49.749350 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:49.750005 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:49.751059 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:49.751965 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:49.752442 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:49.752564 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:49.755276 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:49.755280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:49.755280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:49.755291 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:49.757986 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:49.758456 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:49.758459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:49.758459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:49.758563 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:49.758812 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:49.758814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:49.758814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:49.759157 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:49.759321 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:49.759452 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 17:47:49.759593 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260924 17:47:49.759812 21702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:49.760754 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:49.766417 21702 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:49.767117 21702 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:49.769833 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:49.762321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39855 (local address 127.0.0.1:33343)
0924 17:47:49.762353 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:49.762355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:49.762562 (+ 207us) server_negotiation.cc:407] Connection header received
0924 17:47:49.762596 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:49.762597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:49.762609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:49.762673 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:49.763262 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:49.764076 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:49.765077 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:49.765396 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:49.765472 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:49.767014 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:49.767016 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:49.767017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:49.767025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:49.768637 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:49.769020 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:49.769021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:49.769022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:49.769086 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:49.769291 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:49.769292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:49.769292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:49.769422 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:49.769519 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:49.769629 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:47:49.769723 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 17:47:49.772862 21702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a0385be959348458f6437b56754770f: Generated new TSK 0
I20260924 17:47:49.773051 21702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:49.778071 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21622
2026-09-24T17:47:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:49 dist-test-slave-r7hf krb5kdc[21721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:49 dist-test-slave-r7hf krb5kdc[21721](info): set up 2 sockets
Sep 24 17:47:49 dist-test-slave-r7hf krb5kdc[21721](info): commencing operation
krb5kdc: starting...
W20260924 17:47:50.814651 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:50 dist-test-slave-r7hf krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:50.940766 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:51.026419 21734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:51.026675 21734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:51.026716 21734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:51.027925 21734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:51.028013 21734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:51.028040 21734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:51.028072 21734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:51.028095 21734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:51.029629 21734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:51.029929 21734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:51.030175 21734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:51.035727 21739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:51.035748 21742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:51.035815 21734 server_base.cc:1034] running on GCE node
W20260924 17:47:51.035743 21740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:51.036175 21734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:51.036501 21734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:51.037670 21734 hybrid_clock.cc:648] HybridClock initialized: now 1790272071037666 us; error 39 us; skew 500 ppm
Sep 24 17:47:51 dist-test-slave-r7hf krb5kdc[21721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:51.045265 21734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:51.045849 21734 webserver.cc:460] Webserver started at http://127.0.0.1:38661/ using document root <none> and password file <none>
I20260924 17:47:51.046092 21734 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:51.046161 21734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:51.046315 21734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:51.047503 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "e13135900fb74b50bc091d7f00571835"
format_stamp: "Formatted at 2026-09-24 17:47:51 on dist-test-slave-r7hf"
server_key: "7875b8bc5ddf7aecd4906b7f48d3416a"
server_key_iv: "3936c06061cb660dbb04a103bbc1e0a9"
server_key_version: "encryptionkey@0"
I20260924 17:47:51.047889 21734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "e13135900fb74b50bc091d7f00571835"
format_stamp: "Formatted at 2026-09-24 17:47:51 on dist-test-slave-r7hf"
server_key: "7875b8bc5ddf7aecd4906b7f48d3416a"
server_key_iv: "3936c06061cb660dbb04a103bbc1e0a9"
server_key_version: "encryptionkey@0"
I20260924 17:47:51.049278 21734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:51.050043 21749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:51.050246 21734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:51.050359 21734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "e13135900fb74b50bc091d7f00571835"
format_stamp: "Formatted at 2026-09-24 17:47:51 on dist-test-slave-r7hf"
server_key: "7875b8bc5ddf7aecd4906b7f48d3416a"
server_key_iv: "3936c06061cb660dbb04a103bbc1e0a9"
server_key_version: "encryptionkey@0"
I20260924 17:47:51.050489 21734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:51.088594 21734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:51.090705 21734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:51.090894 21734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:51.095120 21734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37247
I20260924 17:47:51.095153 21801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37247 every 8 connection(s)
I20260924 17:47:51.095520 21734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:51.096606 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21734
I20260924 17:47:51.096616 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:51.096694 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:51.096904 1058 external_mini_cluster.cc:1428] Setting key 525f929677f550c6feba415562f96b40
I20260924 17:47:51.100167 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835: Bootstrap starting.
I20260924 17:47:51.100920 21802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:51.101223 21802 log.cc:824] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:51.101994 21802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835: No bootstrap required, opened a new log
I20260924 17:47:51.103077 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:51.097460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45455 (local address 127.0.0.1:37247)
0924 17:47:51.097640 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:51.097647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:51.097673 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:47:51.098188 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:51.098195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:51.098247 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:51.098370 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:51.098783 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:51.099695 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:51.100808 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:51.101284 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:51.101383 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:51.102215 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:51.102220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:51.102220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:51.102470 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:51.102512 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:51.102527 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:51.102598 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:51.102821 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 17:47:51.102895 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 17:47:51.104151 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:51.104481 21802 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13135900fb74b50bc091d7f00571835" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260924 17:47:51.104650 21802 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:51.104698 21802 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13135900fb74b50bc091d7f00571835, State: Initialized, Role: FOLLOWER
I20260924 17:47:51.104832 21802 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13135900fb74b50bc091d7f00571835" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260924 17:47:51.104931 21802 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:51.104974 21802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:51.105024 21802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:51.105650 21802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13135900fb74b50bc091d7f00571835" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260924 17:47:51.105778 21802 leader_election.cc:302] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13135900fb74b50bc091d7f00571835; no voters:
I20260924 17:47:51.105962 21802 leader_election.cc:288] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:51.106022 21812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:51.106166 21812 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [term 1 LEADER]: Becoming Leader. State: Replica: e13135900fb74b50bc091d7f00571835, State: Running, Role: LEADER
I20260924 17:47:51.106235 21802 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:51.106350 21812 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13135900fb74b50bc091d7f00571835" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } }
I20260924 17:47:51.106786 21812 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13135900fb74b50bc091d7f00571835. Latest consensus state: current_term: 1 leader_uuid: "e13135900fb74b50bc091d7f00571835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13135900fb74b50bc091d7f00571835" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } } }
I20260924 17:47:51.106797 21814 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13135900fb74b50bc091d7f00571835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13135900fb74b50bc091d7f00571835" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37247 } } }
I20260924 17:47:51.106942 21814 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:51.107149 21812 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835 [sys.catalog]: This master's current role is: LEADER
W20260924 17:47:51.108250 21828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:47:51.108441 21828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:47:51.108515 21820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:51.108745 21820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:51.109280 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:51.105568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41179 (local address 127.0.0.1:37247)
0924 17:47:51.105611 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:51.105612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:51.105726 (+ 114us) server_negotiation.cc:407] Connection header received
0924 17:47:51.105797 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:51.105798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:51.105810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:51.105860 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:51.106000 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:51.106772 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:51.107872 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:51.108156 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:51.108247 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:51.109041 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:51.109042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:51.109043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:51.109047 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:51.109063 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:51.109067 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:51.109111 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:51.109188 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 17:47:51.109208 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:47:51.111521 21820 catalog_manager.cc:1347] Generated new cluster ID: d3a0898a265643beab27f373ebb56758
I20260924 17:47:51.111574 21820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:51.133796 21820 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:51.134449 21820 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:51.139010 21820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13135900fb74b50bc091d7f00571835: Generated new TSK 0
I20260924 17:47:51.139247 21820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:51.141912 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21734
2026-09-24T17:47:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:51 dist-test-slave-r7hf krb5kdc[21833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:51 dist-test-slave-r7hf krb5kdc[21833](info): set up 2 sockets
Sep 24 17:47:51 dist-test-slave-r7hf krb5kdc[21833](info): commencing operation
krb5kdc: starting...
W20260924 17:47:52.182870 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:52 dist-test-slave-r7hf krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:52.310246 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:52.393944 21846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:52.394182 21846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:52.394208 21846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:52.395462 21846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:52.395507 21846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:52.395522 21846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:52.395535 21846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:52.395547 21846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:52.397004 21846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:52.397219 21846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:52.397418 21846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:52.402503 21851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:52.402494 21852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:52.402565 21846 server_base.cc:1034] running on GCE node
W20260924 17:47:52.402494 21854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:52.403000 21846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:52.403307 21846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:52.404471 21846 hybrid_clock.cc:648] HybridClock initialized: now 1790272072404461 us; error 35 us; skew 500 ppm
Sep 24 17:47:52 dist-test-slave-r7hf krb5kdc[21833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:52.407156 21846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:52.407557 21846 webserver.cc:460] Webserver started at http://127.0.0.1:45525/ using document root <none> and password file <none>
I20260924 17:47:52.407764 21846 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:52.407806 21846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:52.407895 21846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:52.408919 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "f5598d963ef74f09946ec25f4ba09f4f"
format_stamp: "Formatted at 2026-09-24 17:47:52 on dist-test-slave-r7hf"
server_key: "bef8b3c3fe1e9970797329809ced2963"
server_key_iv: "cf1e92cc9077eaa271861e2ab89a34a1"
server_key_version: "encryptionkey@0"
I20260924 17:47:52.409215 21846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "f5598d963ef74f09946ec25f4ba09f4f"
format_stamp: "Formatted at 2026-09-24 17:47:52 on dist-test-slave-r7hf"
server_key: "bef8b3c3fe1e9970797329809ced2963"
server_key_iv: "cf1e92cc9077eaa271861e2ab89a34a1"
server_key_version: "encryptionkey@0"
I20260924 17:47:52.410528 21846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:47:52.411273 21861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:52.411451 21846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:52.411538 21846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "f5598d963ef74f09946ec25f4ba09f4f"
format_stamp: "Formatted at 2026-09-24 17:47:52 on dist-test-slave-r7hf"
server_key: "bef8b3c3fe1e9970797329809ced2963"
server_key_iv: "cf1e92cc9077eaa271861e2ab89a34a1"
server_key_version: "encryptionkey@0"
I20260924 17:47:52.411597 21846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:52.414897 21846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:52.416875 21846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:52.416983 21846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:52.420882 21846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35691
I20260924 17:47:52.420971 21913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35691 every 8 connection(s)
I20260924 17:47:52.421293 21846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:52.422293 21914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:52.424589 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f: Bootstrap starting.
I20260924 17:47:52.425225 21914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:52.425299 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21846
I20260924 17:47:52.425452 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:52.425586 21914 log.cc:824] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:52.425632 1058 external_mini_cluster.cc:1428] Setting key 94d299e9d434b35a535903aab6c70349
I20260924 17:47:52.426478 21914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f: No bootstrap required, opened a new log
I20260924 17:47:52.428279 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:52.426156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47059 (local address 127.0.0.1:35691)
0924 17:47:52.426356 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:52.426366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:52.426435 (+ 69us) server_negotiation.cc:407] Connection header received
0924 17:47:52.426969 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:52.426978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:52.427014 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:52.427185 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:52.427831 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:52.427835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:52.427835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:52.427840 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:52.427885 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:52.427900 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:52.427993 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:52.428026 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 17:47:52.428067 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 17:47:52.429144 21914 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5598d963ef74f09946ec25f4ba09f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260924 17:47:52.429301 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:52.429307 21914 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:52.429343 21914 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5598d963ef74f09946ec25f4ba09f4f, State: Initialized, Role: FOLLOWER
I20260924 17:47:52.429461 21914 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5598d963ef74f09946ec25f4ba09f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260924 17:47:52.429543 21914 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:52.429569 21914 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:52.429615 21914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:52.430377 21914 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5598d963ef74f09946ec25f4ba09f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260924 17:47:52.430536 21914 leader_election.cc:302] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5598d963ef74f09946ec25f4ba09f4f; no voters:
I20260924 17:47:52.430722 21914 leader_election.cc:288] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:52.430755 21923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:52.430902 21923 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [term 1 LEADER]: Becoming Leader. State: Replica: f5598d963ef74f09946ec25f4ba09f4f, State: Running, Role: LEADER
I20260924 17:47:52.430972 21914 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:52.431002 21923 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5598d963ef74f09946ec25f4ba09f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260924 17:47:52.431346 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5598d963ef74f09946ec25f4ba09f4f. Latest consensus state: current_term: 1 leader_uuid: "f5598d963ef74f09946ec25f4ba09f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5598d963ef74f09946ec25f4ba09f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260924 17:47:52.431432 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:52.431360 21925 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5598d963ef74f09946ec25f4ba09f4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5598d963ef74f09946ec25f4ba09f4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260924 17:47:52.431607 21925 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:52.431679 21931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:52.432186 21931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:52.433578 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:52.431038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33999 (local address 127.0.0.1:35691)
0924 17:47:52.431074 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:52.431075 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:52.432521 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 17:47:52.432621 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:52.432622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:52.432628 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:52.432687 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:52.433344 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:52.433345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:52.433346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:52.433350 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:52.433364 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:52.433368 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:52.433422 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:52.433490 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 17:47:52.433504 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 17:47:52.433965 21931 catalog_manager.cc:1347] Generated new cluster ID: 1ff20bc98437455eb9e79c6a80481e39
I20260924 17:47:52.434017 21931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:52.451771 21931 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:52.451997 21931 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:52.455201 21931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5598d963ef74f09946ec25f4ba09f4f: Generated new TSK 0
I20260924 17:47:52.455422 21931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:52.463903 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21846
2026-09-24T17:47:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:52 dist-test-slave-r7hf krb5kdc[21945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:52 dist-test-slave-r7hf krb5kdc[21945](info): set up 2 sockets
Sep 24 17:47:52 dist-test-slave-r7hf krb5kdc[21945](info): commencing operation
krb5kdc: starting...
W20260924 17:47:53.502657 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:53 dist-test-slave-r7hf krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:53.629328 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:53.712332 21958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:53.712595 21958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:53.712636 21958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:53.713860 21958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:53.713954 21958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:53.713979 21958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:53.714000 21958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:53.714032 21958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:53.715590 21958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:53.715890 21958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:53.716130 21958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:53.721406 21964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:53.721431 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:53.721639 21958 server_base.cc:1034] running on GCE node
W20260924 17:47:53.721429 21963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:53.721958 21958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:53.722224 21958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:53.723366 21958 hybrid_clock.cc:648] HybridClock initialized: now 1790272073723344 us; error 38 us; skew 500 ppm
Sep 24 17:47:53 dist-test-slave-r7hf krb5kdc[21945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:53.725893 21958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:53.726370 21958 webserver.cc:460] Webserver started at http://127.0.0.1:38647/ using document root <none> and password file <none>
I20260924 17:47:53.726679 21958 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:53.726755 21958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:53.726912 21958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:53.728027 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "8e547892274b4c5d918ef8f9036a8914"
format_stamp: "Formatted at 2026-09-24 17:47:53 on dist-test-slave-r7hf"
server_key: "bf35d0ee5e78beaba2346fef9ab531e4"
server_key_iv: "a4558687b0846a9401481fc1a7cea99c"
server_key_version: "encryptionkey@0"
I20260924 17:47:53.728411 21958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "8e547892274b4c5d918ef8f9036a8914"
format_stamp: "Formatted at 2026-09-24 17:47:53 on dist-test-slave-r7hf"
server_key: "bf35d0ee5e78beaba2346fef9ab531e4"
server_key_iv: "a4558687b0846a9401481fc1a7cea99c"
server_key_version: "encryptionkey@0"
I20260924 17:47:53.729812 21958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:47:53.730602 21973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:53.730808 21958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:53.730921 21958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "8e547892274b4c5d918ef8f9036a8914"
format_stamp: "Formatted at 2026-09-24 17:47:53 on dist-test-slave-r7hf"
server_key: "bf35d0ee5e78beaba2346fef9ab531e4"
server_key_iv: "a4558687b0846a9401481fc1a7cea99c"
server_key_version: "encryptionkey@0"
I20260924 17:47:53.731000 21958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:53.753765 21958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:53.755882 21958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:53.756063 21958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:53.760493 21958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32799
I20260924 17:47:53.760531 22025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32799 every 8 connection(s)
I20260924 17:47:53.760921 21958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:53.762105 22026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:53.764760 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914: Bootstrap starting.
I20260924 17:47:53.764778 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 21958
I20260924 17:47:53.764899 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:53.765096 1058 external_mini_cluster.cc:1428] Setting key 951ffac474529481881e45c5b09f1bce
I20260924 17:47:53.765516 22026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:53.765923 22026 log.cc:824] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:53.766724 22026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914: No bootstrap required, opened a new log
I20260924 17:47:53.769142 22026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e547892274b4c5d918ef8f9036a8914" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32799 } }
I20260924 17:47:53.769323 22026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:53.769354 22026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e547892274b4c5d918ef8f9036a8914, State: Initialized, Role: FOLLOWER
I20260924 17:47:53.769471 22026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e547892274b4c5d918ef8f9036a8914" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32799 } }
I20260924 17:47:53.769549 22026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:53.769590 22026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:53.769654 22026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:53.770268 22026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e547892274b4c5d918ef8f9036a8914" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32799 } }
I20260924 17:47:53.770414 22026 leader_election.cc:302] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e547892274b4c5d918ef8f9036a8914; no voters:
I20260924 17:47:53.770601 22026 leader_election.cc:288] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:53.770654 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:53.770836 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [term 1 LEADER]: Becoming Leader. State: Replica: 8e547892274b4c5d918ef8f9036a8914, State: Running, Role: LEADER
I20260924 17:47:53.770890 22026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:53.770954 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e547892274b4c5d918ef8f9036a8914" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32799 } }
I20260924 17:47:53.771360 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e547892274b4c5d918ef8f9036a8914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e547892274b4c5d918ef8f9036a8914" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32799 } } }
I20260924 17:47:53.771353 22031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e547892274b4c5d918ef8f9036a8914. Latest consensus state: current_term: 1 leader_uuid: "8e547892274b4c5d918ef8f9036a8914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e547892274b4c5d918ef8f9036a8914" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32799 } } }
I20260924 17:47:53.771510 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:53.771520 22031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:53.772686 22040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 17:47:53 dist-test-slave-r7hf krb5kdc[21945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:47:53.773201 22040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:53.774931 22040 catalog_manager.cc:1347] Generated new cluster ID: 7bbd5045409a4a13a21d295778c9d7a5
I20260924 17:47:53.775015 22040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:53.777326 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:53.765694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:47963 (local address 127.0.0.1:32799)
0924 17:47:53.765886 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:53.765896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:53.765934 (+ 38us) server_negotiation.cc:407] Connection header received
0924 17:47:53.766592 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:53.766601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:53.766659 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:53.766810 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:53.767589 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:53.768495 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:53.769597 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:53.770090 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:53.770182 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:53.773401 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:53.773405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:53.773405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:53.773418 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:53.775554 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:53.776014 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:53.776017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:53.776018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:53.776128 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:53.776377 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:53.776378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:53.776379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:53.776695 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:53.776883 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:53.776977 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 17:47:53.777087 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260924 17:47:53.778350 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:53.787911 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:53.779916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:49279 (local address 127.0.0.1:32799)
0924 17:47:53.779959 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:53.779961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:53.780058 (+ 97us) server_negotiation.cc:407] Connection header received
0924 17:47:53.780118 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:53.780119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:53.780130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:53.780173 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:53.780812 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:53.781904 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:53.782979 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:53.783305 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:53.783366 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:53.784881 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:53.784882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:53.784882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:53.784890 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:53.786534 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:53.786960 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:53.786961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:53.786961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:53.787020 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:53.787260 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:53.787261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:53.787261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:53.787382 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:53.787490 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:53.787662 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 17:47:53.787724 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 17:47:53.800748 22040 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:53.801321 22040 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:53.807744 22040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e547892274b4c5d918ef8f9036a8914: Generated new TSK 0
I20260924 17:47:53.807977 22040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:53.812625 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 21958
W20260924 17:47:53.817114 21943 connection.cc:535] client connection to 127.0.0.1:32799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32799): Connection reset by peer (error 104)
2026-09-24T17:47:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:53 dist-test-slave-r7hf krb5kdc[22057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:53 dist-test-slave-r7hf krb5kdc[22057](info): set up 2 sockets
Sep 24 17:47:53 dist-test-slave-r7hf krb5kdc[22057](info): commencing operation
krb5kdc: starting...
W20260924 17:47:54.850545 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:54 dist-test-slave-r7hf krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:54.971545 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:55.054179 22070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:55.054437 22070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:55.054463 22070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:55.055691 22070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:55.055740 22070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:55.055754 22070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:55.055768 22070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:55.055824 22070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:55.057319 22070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:55.057590 22070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:55.057832 22070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:55.063133 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:55.063235 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:55.063349 22070 server_base.cc:1034] running on GCE node
W20260924 17:47:55.063133 22078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:55.063658 22070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:55.063956 22070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:55.065106 22070 hybrid_clock.cc:648] HybridClock initialized: now 1790272075065086 us; error 40 us; skew 500 ppm
Sep 24 17:47:55 dist-test-slave-r7hf krb5kdc[22057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:55.067624 22070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:55.068125 22070 webserver.cc:460] Webserver started at http://127.0.0.1:35897/ using document root <none> and password file <none>
I20260924 17:47:55.068383 22070 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:55.068454 22070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:55.068602 22070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:55.069676 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "25ca11ead8174d7dab246c6bfea0bc6d"
format_stamp: "Formatted at 2026-09-24 17:47:55 on dist-test-slave-r7hf"
server_key: "2ddb224b0623cacbbdd51779c7572d91"
server_key_iv: "7a16c33652badd0673b4bdd39bf5e886"
server_key_version: "encryptionkey@0"
I20260924 17:47:55.070039 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "25ca11ead8174d7dab246c6bfea0bc6d"
format_stamp: "Formatted at 2026-09-24 17:47:55 on dist-test-slave-r7hf"
server_key: "2ddb224b0623cacbbdd51779c7572d91"
server_key_iv: "7a16c33652badd0673b4bdd39bf5e886"
server_key_version: "encryptionkey@0"
I20260924 17:47:55.071450 22070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260924 17:47:55.072216 22085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:55.072386 22070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:55.072501 22070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "25ca11ead8174d7dab246c6bfea0bc6d"
format_stamp: "Formatted at 2026-09-24 17:47:55 on dist-test-slave-r7hf"
server_key: "2ddb224b0623cacbbdd51779c7572d91"
server_key_iv: "7a16c33652badd0673b4bdd39bf5e886"
server_key_version: "encryptionkey@0"
I20260924 17:47:55.072578 22070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:55.101261 22070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:55.103314 22070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:55.103492 22070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:55.107620 22070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46733
I20260924 17:47:55.107669 22137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46733 every 8 connection(s)
I20260924 17:47:55.108047 22070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:55.109038 22138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:55.111436 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d: Bootstrap starting.
I20260924 17:47:55.111990 22138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:55.112371 22138 log.cc:824] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:55.113109 22138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d: No bootstrap required, opened a new log
I20260924 17:47:55.114782 22138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46733 } }
I20260924 17:47:55.114929 22138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:55.114952 22138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ca11ead8174d7dab246c6bfea0bc6d, State: Initialized, Role: FOLLOWER
I20260924 17:47:55.115156 22138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46733 } }
I20260924 17:47:55.115240 22138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:55.115286 22138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:55.115338 22138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:55.115881 22138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46733 } }
I20260924 17:47:55.115998 22138 leader_election.cc:302] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25ca11ead8174d7dab246c6bfea0bc6d; no voters:
I20260924 17:47:55.116199 22138 leader_election.cc:288] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:55.116315 22141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:55.116479 22138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:55.116557 22141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [term 1 LEADER]: Becoming Leader. State: Replica: 25ca11ead8174d7dab246c6bfea0bc6d, State: Running, Role: LEADER
I20260924 17:47:55.116662 22141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46733 } }
I20260924 17:47:55.117133 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22070
I20260924 17:47:55.117264 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:55.117317 22143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25ca11ead8174d7dab246c6bfea0bc6d. Latest consensus state: current_term: 1 leader_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46733 } } }
I20260924 17:47:55.117390 22143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:55.117517 1058 external_mini_cluster.cc:1428] Setting key 07f108612c09e0e197ff3d53ed7d07bb
I20260924 17:47:55.117503 22142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25ca11ead8174d7dab246c6bfea0bc6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46733 } } }
I20260924 17:47:55.117569 22142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:55.117631 22150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:55.118309 22150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:55.120164 22150 catalog_manager.cc:1347] Generated new cluster ID: 19f05e7832134e3e9b5edd8a7e6740be
I20260924 17:47:55.120223 22150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:55.123448 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:55.118123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:38061 (local address 127.0.0.1:46733)
0924 17:47:55.118420 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:55.118431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:55.118466 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:47:55.118779 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:55.118789 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:55.118852 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:55.118999 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:55.119329 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:55.120619 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:55.121428 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:55.121889 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:55.122015 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:55.122792 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:55.122796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:55.122796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:55.122980 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:55.123010 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:55.123022 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:55.123116 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:55.123202 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 17:47:55.123233 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":65,"threads_started":1}
I20260924 17:47:55.124362 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:55.128696 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:55.125761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:37663 (local address 127.0.0.1:46733)
0924 17:47:55.125798 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:55.125800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:55.125910 (+ 110us) server_negotiation.cc:407] Connection header received
0924 17:47:55.125978 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:55.125979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:55.125990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:55.126035 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:55.126150 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:55.126716 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:55.127419 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:55.127636 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:55.127714 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:55.128418 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:55.128419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:55.128420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:55.128429 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:55.128443 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:55.128448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:55.128520 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:55.128586 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 17:47:55.128598 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:47:55.131896 22150 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:55.132601 22150 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:55.146000 22150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25ca11ead8174d7dab246c6bfea0bc6d: Generated new TSK 0
I20260924 17:47:55.146250 22150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:55.148989 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22070
W20260924 17:47:55.153178 22055 connection.cc:535] client connection to 127.0.0.1:46733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46733): Connection reset by peer (error 104)
2026-09-24T17:47:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:55 dist-test-slave-r7hf krb5kdc[22169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:55 dist-test-slave-r7hf krb5kdc[22169](info): set up 2 sockets
Sep 24 17:47:55 dist-test-slave-r7hf krb5kdc[22169](info): commencing operation
krb5kdc: starting...
W20260924 17:47:56.190625 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:56 dist-test-slave-r7hf krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:56.315315 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:56.397284 22182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:56.397513 22182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:56.397538 22182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:56.398840 22182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:56.398890 22182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:56.398904 22182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:56.398952 22182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:56.398983 22182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:56.400498 22182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:56.400763 22182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:56.400996 22182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:56.406167 22187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:56.406195 22182 server_base.cc:1034] running on GCE node
W20260924 17:47:56.406239 22188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:56.406159 22190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:56.406610 22182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:56.406878 22182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:56.408005 22182 hybrid_clock.cc:648] HybridClock initialized: now 1790272076407987 us; error 35 us; skew 500 ppm
Sep 24 17:47:56 dist-test-slave-r7hf krb5kdc[22169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:56.414386 22182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:56.414928 22182 webserver.cc:460] Webserver started at http://127.0.0.1:45941/ using document root <none> and password file <none>
I20260924 17:47:56.415267 22182 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:56.415321 22182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:56.415475 22182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:56.416596 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "b966ea8bfe504fcc99c6a706e8877562"
format_stamp: "Formatted at 2026-09-24 17:47:56 on dist-test-slave-r7hf"
server_key: "90a86ee1b4934b83a8ca47c85b092204"
server_key_iv: "f129d626601bf11cc48c5ed1541ee9ca"
server_key_version: "encryptionkey@0"
I20260924 17:47:56.416970 22182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "b966ea8bfe504fcc99c6a706e8877562"
format_stamp: "Formatted at 2026-09-24 17:47:56 on dist-test-slave-r7hf"
server_key: "90a86ee1b4934b83a8ca47c85b092204"
server_key_iv: "f129d626601bf11cc48c5ed1541ee9ca"
server_key_version: "encryptionkey@0"
I20260924 17:47:56.418354 22182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 17:47:56.419193 22197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:56.419384 22182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:56.419483 22182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "b966ea8bfe504fcc99c6a706e8877562"
format_stamp: "Formatted at 2026-09-24 17:47:56 on dist-test-slave-r7hf"
server_key: "90a86ee1b4934b83a8ca47c85b092204"
server_key_iv: "f129d626601bf11cc48c5ed1541ee9ca"
server_key_version: "encryptionkey@0"
I20260924 17:47:56.419560 22182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:56.423050 22182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:56.425000 22182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:56.425164 22182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:56.429171 22182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46783
I20260924 17:47:56.429201 22249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46783 every 8 connection(s)
I20260924 17:47:56.429548 22182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:56.430006 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22182
I20260924 17:47:56.430167 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:56.430438 1058 external_mini_cluster.cc:1428] Setting key ba8244cb9eb961a982e06de27123082e
I20260924 17:47:56.430868 22250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:56.433254 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:56.431026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:46651 (local address 127.0.0.1:46783)
0924 17:47:56.431230 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:56.431246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:56.431271 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:47:56.431944 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:56.431952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:56.431981 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:56.432105 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:56.432776 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:56.432780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:56.432780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:56.432784 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:56.432826 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:56.432837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:56.432895 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:56.432999 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 17:47:56.433065 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260924 17:47:56.434250 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:56.434397 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562: Bootstrap starting.
I20260924 17:47:56.435196 22250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:56.435570 22250 log.cc:824] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:56.436326 22250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562: No bootstrap required, opened a new log
I20260924 17:47:56.438019 22250 raft_consensus.cc:348] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b966ea8bfe504fcc99c6a706e8877562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260924 17:47:56.438166 22250 raft_consensus.cc:374] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:56.438190 22250 raft_consensus.cc:729] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b966ea8bfe504fcc99c6a706e8877562, State: Initialized, Role: FOLLOWER
I20260924 17:47:56.438277 22250 consensus_queue.cc:260] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b966ea8bfe504fcc99c6a706e8877562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260924 17:47:56.438335 22250 raft_consensus.cc:388] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:56.438355 22250 raft_consensus.cc:482] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:56.438383 22250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:56.439034 22250 raft_consensus.cc:504] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b966ea8bfe504fcc99c6a706e8877562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260924 17:47:56.439152 22250 leader_election.cc:302] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b966ea8bfe504fcc99c6a706e8877562; no voters:
I20260924 17:47:56.439321 22250 leader_election.cc:288] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:56.439374 22260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:56.439594 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:56.435883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:55735 (local address 127.0.0.1:46783)
0924 17:47:56.435949 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:56.435951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:56.438466 (+ 2515us) server_negotiation.cc:407] Connection header received
0924 17:47:56.438494 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:56.438495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:56.438501 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:56.438570 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:56.439205 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:56.439206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:56.439207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:56.439211 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:56.439225 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:56.439230 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:56.439275 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:56.439355 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 17:47:56.439369 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 17:47:56.439654 22250 sys_catalog.cc:564] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:56.439746 22260 raft_consensus.cc:686] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [term 1 LEADER]: Becoming Leader. State: Replica: b966ea8bfe504fcc99c6a706e8877562, State: Running, Role: LEADER
I20260924 17:47:56.439846 22260 consensus_queue.cc:237] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b966ea8bfe504fcc99c6a706e8877562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260924 17:47:56.440511 22260 sys_catalog.cc:455] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b966ea8bfe504fcc99c6a706e8877562. Latest consensus state: current_term: 1 leader_uuid: "b966ea8bfe504fcc99c6a706e8877562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b966ea8bfe504fcc99c6a706e8877562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } } }
I20260924 17:47:56.440632 22260 sys_catalog.cc:458] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:56.440699 22261 sys_catalog.cc:455] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b966ea8bfe504fcc99c6a706e8877562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b966ea8bfe504fcc99c6a706e8877562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } } }
I20260924 17:47:56.440797 22261 sys_catalog.cc:458] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:56.440892 22270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:56.441447 22270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:56.443027 22270 catalog_manager.cc:1347] Generated new cluster ID: 47fbf11b0ba745cd875cf08f35f46c10
I20260924 17:47:56.443090 22270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:56.455106 22270 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:56.455291 22270 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:56.468590 22270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b966ea8bfe504fcc99c6a706e8877562: Generated new TSK 0
I20260924 17:47:56.468815 22270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:56.475677 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22182
2026-09-24T17:47:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:56 dist-test-slave-r7hf krb5kdc[22281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:56 dist-test-slave-r7hf krb5kdc[22281](info): set up 2 sockets
Sep 24 17:47:56 dist-test-slave-r7hf krb5kdc[22281](info): commencing operation
krb5kdc: starting...
W20260924 17:47:57.506655 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:57 dist-test-slave-r7hf krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:57.630507 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:57.714609 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:57.714831 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:57.714857 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:57.715966 22294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:57.716007 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:57.716024 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:57.716039 22294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:57.716053 22294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:57.718083 22294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:57.718429 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:57.718659 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:57.723802 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:57.723802 22299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:57.723913 22294 server_base.cc:1034] running on GCE node
W20260924 17:47:57.723832 22300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:57.724310 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:57.724623 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:57.725793 22294 hybrid_clock.cc:648] HybridClock initialized: now 1790272077725783 us; error 41 us; skew 500 ppm
Sep 24 17:47:57 dist-test-slave-r7hf krb5kdc[22281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:57.728221 22294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:57.728718 22294 webserver.cc:460] Webserver started at http://127.0.0.1:34775/ using document root <none> and password file <none>
I20260924 17:47:57.728960 22294 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:57.729030 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:57.729185 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:57.730297 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "ce7b38c3239e4f9b840b760f8e9ee279"
format_stamp: "Formatted at 2026-09-24 17:47:57 on dist-test-slave-r7hf"
server_key: "46991a085e435407b03c3b68f7e2b915"
server_key_iv: "5a349f3e5d64a54f38fc8caa26db6ccd"
server_key_version: "encryptionkey@0"
I20260924 17:47:57.730746 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "ce7b38c3239e4f9b840b760f8e9ee279"
format_stamp: "Formatted at 2026-09-24 17:47:57 on dist-test-slave-r7hf"
server_key: "46991a085e435407b03c3b68f7e2b915"
server_key_iv: "5a349f3e5d64a54f38fc8caa26db6ccd"
server_key_version: "encryptionkey@0"
I20260924 17:47:57.732075 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:47:57.732791 22309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:57.732955 22294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:47:57.733062 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "ce7b38c3239e4f9b840b760f8e9ee279"
format_stamp: "Formatted at 2026-09-24 17:47:57 on dist-test-slave-r7hf"
server_key: "46991a085e435407b03c3b68f7e2b915"
server_key_iv: "5a349f3e5d64a54f38fc8caa26db6ccd"
server_key_version: "encryptionkey@0"
I20260924 17:47:57.733191 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:57.753764 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:57.755925 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:57.756100 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:57.760321 22294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41889
I20260924 17:47:57.760357 22361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41889 every 8 connection(s)
I20260924 17:47:57.760747 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:57.761739 22362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:57.764050 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279: Bootstrap starting.
I20260924 17:47:57.764580 22362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:57.764808 22362 log.cc:824] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:57.765431 22362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279: No bootstrap required, opened a new log
I20260924 17:47:57.765709 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22294
I20260924 17:47:57.765828 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:57.766026 1058 external_mini_cluster.cc:1428] Setting key 6cb3302274697e2d9a161142ddc8933f
I20260924 17:47:57.767577 22362 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260924 17:47:57.767757 22362 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:57.767797 22362 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7b38c3239e4f9b840b760f8e9ee279, State: Initialized, Role: FOLLOWER
I20260924 17:47:57.767937 22362 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260924 17:47:57.768028 22362 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:57.768060 22362 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:57.768110 22362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:57.769106 22362 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260924 17:47:57.769197 22362 leader_election.cc:302] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce7b38c3239e4f9b840b760f8e9ee279; no voters:
I20260924 17:47:57.769381 22362 leader_election.cc:288] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:57.769421 22367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:57.769587 22367 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [term 1 LEADER]: Becoming Leader. State: Replica: ce7b38c3239e4f9b840b760f8e9ee279, State: Running, Role: LEADER
I20260924 17:47:57.769646 22362 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:57.769763 22367 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } }
I20260924 17:47:57.770414 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7b38c3239e4f9b840b760f8e9ee279. Latest consensus state: current_term: 1 leader_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } } }
I20260924 17:47:57.770499 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:57.770381 22368 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce7b38c3239e4f9b840b760f8e9ee279" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41889 } } }
I20260924 17:47:57.770771 22368 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:57.770692 22376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:57.771468 22376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 17:47:57 dist-test-slave-r7hf krb5kdc[22281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 17:47:57.774072 22376 catalog_manager.cc:1347] Generated new cluster ID: 31b898e2474c4e929f8dd573c4c039da
I20260924 17:47:57.774127 22376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:57.778263 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:57.766552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:35429 (local address 127.0.0.1:41889)
0924 17:47:57.766750 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:57.766758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:57.766783 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:47:57.767551 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:57.767557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:57.767605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:57.767721 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:57.768568 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:57.769707 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:57.770863 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:57.771489 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:57.771952 (+ 463us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:57.774555 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:57.774559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:57.774559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:57.774579 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:57.776319 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:57.776752 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:57.776755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:57.776755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:57.776858 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:57.777116 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:57.777122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:57.777122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:57.777355 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:57.777962 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:57.778033 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 17:47:57.778117 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
I20260924 17:47:57.779254 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:57.784683 22376 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:57.785362 22376 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:57.787470 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:57.780635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:56857 (local address 127.0.0.1:41889)
0924 17:47:57.780672 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:57.780674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:57.780800 (+ 126us) server_negotiation.cc:407] Connection header received
0924 17:47:57.780863 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:57.780864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:57.780874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:57.780945 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:57.781462 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:57.782006 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:57.782788 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:57.783012 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:57.783098 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:57.784510 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:57.784511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:57.784511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:47:57.784518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:57.786187 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:57.786631 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:57.786633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:57.786633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:57.786694 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:47:57.786924 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:47:57.786925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:47:57.786925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:47:57.787052 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:57.787156 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:57.787313 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 17:47:57.787380 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:47:57.791252 22376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce7b38c3239e4f9b840b760f8e9ee279: Generated new TSK 0
I20260924 17:47:57.791455 22376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:57.795595 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22294
W20260924 17:47:57.799707 22279 connection.cc:535] client connection to 127.0.0.1:41889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41889): Connection reset by peer (error 104)
2026-09-24T17:47:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:57 dist-test-slave-r7hf krb5kdc[22393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:57 dist-test-slave-r7hf krb5kdc[22393](info): set up 2 sockets
Sep 24 17:47:57 dist-test-slave-r7hf krb5kdc[22393](info): commencing operation
krb5kdc: starting...
W20260924 17:47:58.846814 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:47:58 dist-test-slave-r7hf krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:47:58.973798 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:47:59.069845 22406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:47:59.070065 22406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:47:59.070091 22406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:47:59.072027 22406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:47:59.072098 22406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:47:59.072121 22406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:47:59.072140 22406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:47:59.072161 22406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:47:59.074739 22406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:47:59.075071 22406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:47:59.075349 22406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:47:59.082055 22411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:47:59.082079 22414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:59.082197 22406 server_base.cc:1034] running on GCE node
W20260924 17:47:59.082441 22412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:47:59.082856 22406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:47:59.083294 22406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:47:59.084515 22406 hybrid_clock.cc:648] HybridClock initialized: now 1790272079084482 us; error 45 us; skew 500 ppm
Sep 24 17:47:59 dist-test-slave-r7hf krb5kdc[22393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:47:59.088181 22406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:47:59.088881 22406 webserver.cc:460] Webserver started at http://127.0.0.1:38089/ using document root <none> and password file <none>
I20260924 17:47:59.089192 22406 fs_manager.cc:359] Metadata directory not provided
I20260924 17:47:59.089248 22406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:47:59.089356 22406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:47:59.090539 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "68bda62205b64656abf4c977cbec6da8"
format_stamp: "Formatted at 2026-09-24 17:47:59 on dist-test-slave-r7hf"
server_key: "2347f6b4922070549b4f4a44b697f7b5"
server_key_iv: "ddad89abac0d3e3a595c110a5cad6464"
server_key_version: "encryptionkey@0"
I20260924 17:47:59.090889 22406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "68bda62205b64656abf4c977cbec6da8"
format_stamp: "Formatted at 2026-09-24 17:47:59 on dist-test-slave-r7hf"
server_key: "2347f6b4922070549b4f4a44b697f7b5"
server_key_iv: "ddad89abac0d3e3a595c110a5cad6464"
server_key_version: "encryptionkey@0"
I20260924 17:47:59.092346 22406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:59.093266 22421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:47:59.093497 22406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:47:59.093611 22406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "68bda62205b64656abf4c977cbec6da8"
format_stamp: "Formatted at 2026-09-24 17:47:59 on dist-test-slave-r7hf"
server_key: "2347f6b4922070549b4f4a44b697f7b5"
server_key_iv: "ddad89abac0d3e3a595c110a5cad6464"
server_key_version: "encryptionkey@0"
I20260924 17:47:59.093714 22406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:47:59.101203 22406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:47:59.103538 22406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:47:59.103741 22406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:47:59.107820 22406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37025
I20260924 17:47:59.107858 22473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37025 every 8 connection(s)
I20260924 17:47:59.108215 22406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:47:59.109120 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22406
I20260924 17:47:59.109221 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:47:59.109246 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:47:59.109468 1058 external_mini_cluster.cc:1428] Setting key 096ddc9eb80a5a7eb165606e9cbddd9f
I20260924 17:47:59.112823 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8: Bootstrap starting.
I20260924 17:47:59.113451 22474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8: Neither blocks nor log segments found. Creating new log.
I20260924 17:47:59.113758 22474 log.cc:824] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8: Log is configured to *not* fsync() on all Append() calls
I20260924 17:47:59.114574 22474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8: No bootstrap required, opened a new log
I20260924 17:47:59.115991 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:59.110026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:60583 (local address 127.0.0.1:37025)
0924 17:47:59.110276 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:59.110287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:59.110320 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:47:59.110862 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:59.110872 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:59.110946 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:59.111122 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:59.111321 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:59.112676 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:59.113678 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:59.114159 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:59.114337 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:59.115179 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:59.115184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:59.115184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:59.115362 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:59.115395 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:59.115407 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:59.115505 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:59.115632 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 17:47:59.115662 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":132,"threads_started":1}
I20260924 17:47:59.116137 22474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bda62205b64656abf4c977cbec6da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260924 17:47:59.116344 22474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:47:59.116384 22474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bda62205b64656abf4c977cbec6da8, State: Initialized, Role: FOLLOWER
I20260924 17:47:59.116504 22474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bda62205b64656abf4c977cbec6da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260924 17:47:59.116626 22474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:47:59.116660 22474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:47:59.116704 22474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:47:59.117031 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:47:59.117414 22474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bda62205b64656abf4c977cbec6da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260924 17:47:59.117530 22474 leader_election.cc:302] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bda62205b64656abf4c977cbec6da8; no voters:
I20260924 17:47:59.117725 22474 leader_election.cc:288] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:47:59.117755 22479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:47:59.117871 22479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [term 1 LEADER]: Becoming Leader. State: Replica: 68bda62205b64656abf4c977cbec6da8, State: Running, Role: LEADER
I20260924 17:47:59.117986 22474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:47:59.118006 22479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bda62205b64656abf4c977cbec6da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } }
I20260924 17:47:59.118425 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bda62205b64656abf4c977cbec6da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bda62205b64656abf4c977cbec6da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } } }
I20260924 17:47:59.118513 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [sys.catalog]: This master's current role is: LEADER
I20260924 17:47:59.118654 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bda62205b64656abf4c977cbec6da8. Latest consensus state: current_term: 1 leader_uuid: "68bda62205b64656abf4c977cbec6da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bda62205b64656abf4c977cbec6da8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37025 } } }
I20260924 17:47:59.118712 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8 [sys.catalog]: This master's current role is: LEADER
W20260924 17:47:59.119853 22500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:47:59.119971 22500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:47:59.120101 22490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:47:59.120294 22490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:47:59.122051 22490 catalog_manager.cc:1347] Generated new cluster ID: bdfcb99f99bb48de978c4821be491684
I20260924 17:47:59.122131 22490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:47:59.123075 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:47:59.118627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:50111 (local address 127.0.0.1:37025)
0924 17:47:59.118663 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 17:47:59.118665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:47:59.119683 (+ 1018us) server_negotiation.cc:407] Connection header received
0924 17:47:59.119744 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:47:59.119745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:47:59.119757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:47:59.119811 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:47:59.119947 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:59.120666 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:59.121452 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:47:59.121861 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:47:59.121933 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:47:59.122806 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:47:59.122808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:47:59.122808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:47:59.122813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:47:59.122828 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:47:59.122833 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:47:59.122890 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 17:47:59.122988 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 17:47:59.122998 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:47:59.145112 22490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:47:59.145898 22490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:47:59.149644 22490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68bda62205b64656abf4c977cbec6da8: Generated new TSK 0
I20260924 17:47:59.149861 22490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:47:59.159510 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22406
W20260924 17:47:59.164011 22391 connection.cc:535] client connection to 127.0.0.1:37025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37025): Connection reset by peer (error 104)
2026-09-24T17:47:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:47:59 dist-test-slave-r7hf krb5kdc[22505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:47:59 dist-test-slave-r7hf krb5kdc[22505](info): set up 2 sockets
Sep 24 17:47:59 dist-test-slave-r7hf krb5kdc[22505](info): commencing operation
krb5kdc: starting...
W20260924 17:48:00.226663 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:00 dist-test-slave-r7hf krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 17:48:00.353571 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:00.436547 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:00.436766 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:00.436789 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:00.437881 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:00.437919 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:00.437934 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:00.437947 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:00.437959 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:00.439428 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45075
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:00.439667 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:00.439860 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:00.444813 22524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:00.445008 22526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:00.444981 22523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:00.445199 22518 server_base.cc:1034] running on GCE node
I20260924 17:48:00.445482 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:00.445820 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:00.447013 22518 hybrid_clock.cc:648] HybridClock initialized: now 1790272080446987 us; error 35 us; skew 500 ppm
Sep 24 17:48:00 dist-test-slave-r7hf krb5kdc[22505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:00.449687 22518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 17:48:00.450291 22518 webserver.cc:460] Webserver started at http://127.0.0.1:43587/ using document root <none> and password file <none>
I20260924 17:48:00.450583 22518 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:00.450672 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:00.450840 22518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:00.452054 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "bb6337de3dbd431b9ebc7c9146db755b"
format_stamp: "Formatted at 2026-09-24 17:48:00 on dist-test-slave-r7hf"
server_key: "9ca99eecbaad700932fbab0c00e24dae"
server_key_iv: "4364916aaa58a232430123d02ae6d3fa"
server_key_version: "encryptionkey@0"
I20260924 17:48:00.452461 22518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "bb6337de3dbd431b9ebc7c9146db755b"
format_stamp: "Formatted at 2026-09-24 17:48:00 on dist-test-slave-r7hf"
server_key: "9ca99eecbaad700932fbab0c00e24dae"
server_key_iv: "4364916aaa58a232430123d02ae6d3fa"
server_key_version: "encryptionkey@0"
I20260924 17:48:00.453996 22518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:48:00.454905 22533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:00.455124 22518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 17:48:00.455229 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "bb6337de3dbd431b9ebc7c9146db755b"
format_stamp: "Formatted at 2026-09-24 17:48:00 on dist-test-slave-r7hf"
server_key: "9ca99eecbaad700932fbab0c00e24dae"
server_key_iv: "4364916aaa58a232430123d02ae6d3fa"
server_key_version: "encryptionkey@0"
I20260924 17:48:00.455315 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:00.458966 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:00.461130 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:00.461323 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:00.465961 22518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45075
I20260924 17:48:00.465986 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45075 every 8 connection(s)
I20260924 17:48:00.466378 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:00.467532 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:00.468375 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22518
I20260924 17:48:00.468525 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:00.468724 1058 external_mini_cluster.cc:1428] Setting key b683b4c690875a2318d181262ac86784
I20260924 17:48:00.470527 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b: Bootstrap starting.
I20260924 17:48:00.471156 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:00.471465 22586 log.cc:824] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:00.471865 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:00.469343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:48635 (local address 127.0.0.1:45075)
0924 17:48:00.469540 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:00.469551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:00.469613 (+ 62us) server_negotiation.cc:407] Connection header received
0924 17:48:00.470211 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:00.470220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:00.470255 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:00.470425 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:00.471348 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:00.471352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:00.471352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:00.471356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:00.471405 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:00.471416 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:00.471505 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:00.471587 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 17:48:00.471637 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260924 17:48:00.472271 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b: No bootstrap required, opened a new log
I20260924 17:48:00.472860 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:00.475363 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6337de3dbd431b9ebc7c9146db755b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260924 17:48:00.475569 22586 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:00.475646 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6337de3dbd431b9ebc7c9146db755b, State: Initialized, Role: FOLLOWER
I20260924 17:48:00.475881 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:00.474613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:50969 (local address 127.0.0.1:45075)
0924 17:48:00.474674 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:00.474676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:00.474821 (+ 145us) server_negotiation.cc:407] Connection header received
0924 17:48:00.474902 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:00.474903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:00.474906 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:00.474950 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:00.475527 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:00.475528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:00.475528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:00.475532 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:00.475545 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:00.475550 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:00.475629 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:00.475781 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 17:48:00.475795 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:48:00.475805 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6337de3dbd431b9ebc7c9146db755b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260924 17:48:00.475953 22586 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:00.475999 22586 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:00.476059 22586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:00.476778 22586 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6337de3dbd431b9ebc7c9146db755b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260924 17:48:00.476873 22586 leader_election.cc:302] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb6337de3dbd431b9ebc7c9146db755b; no voters:
I20260924 17:48:00.477075 22586 leader_election.cc:288] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:00.477133 22596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:00.477283 22596 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [term 1 LEADER]: Becoming Leader. State: Replica: bb6337de3dbd431b9ebc7c9146db755b, State: Running, Role: LEADER
I20260924 17:48:00.477366 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:00.477371 22596 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6337de3dbd431b9ebc7c9146db755b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } }
I20260924 17:48:00.477807 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb6337de3dbd431b9ebc7c9146db755b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6337de3dbd431b9ebc7c9146db755b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } } }
I20260924 17:48:00.477855 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb6337de3dbd431b9ebc7c9146db755b. Latest consensus state: current_term: 1 leader_uuid: "bb6337de3dbd431b9ebc7c9146db755b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb6337de3dbd431b9ebc7c9146db755b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45075 } } }
I20260924 17:48:00.477913 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:00.478152 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:00.478206 22604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:00.478729 22604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:00.480443 22604 catalog_manager.cc:1347] Generated new cluster ID: 2fa5ca7986bd4f35a2e1bc34ea18a59c
I20260924 17:48:00.480513 22604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:00.490336 22604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:00.490613 22604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:00.494294 22604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb6337de3dbd431b9ebc7c9146db755b: Generated new TSK 0
I20260924 17:48:00.494526 22604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:00.499285 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22518
2026-09-24T17:48:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:00 dist-test-slave-r7hf krb5kdc[22617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:00 dist-test-slave-r7hf krb5kdc[22617](info): set up 2 sockets
Sep 24 17:48:00 dist-test-slave-r7hf krb5kdc[22617](info): commencing operation
krb5kdc: starting...
W20260924 17:48:01.582692 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:01 dist-test-slave-r7hf krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:01.705675 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45071
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:44937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:01.787204 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:01.787472 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:01.787581 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:01.788817 22630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:01.788897 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:01.788924 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:01.788956 22630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:01.788983 22630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:01.790591 22630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:44937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:01.790879 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:01.791115 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:01.796264 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:01.796268 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:01.796360 22630 server_base.cc:1034] running on GCE node
W20260924 17:48:01.796283 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:01.796777 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:01.797029 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:01.798213 22630 hybrid_clock.cc:648] HybridClock initialized: now 1790272081798191 us; error 53 us; skew 500 ppm
Sep 24 17:48:01 dist-test-slave-r7hf krb5kdc[22617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272081, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:01.805405 22630 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:01.805891 22630 webserver.cc:460] Webserver started at http://127.1.8.190:41853/ using document root <none> and password file <none>
I20260924 17:48:01.806142 22630 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:01.806216 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:01.806368 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:01.807507 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "3d0b6cda972f47229576f2d82b8fad2d"
format_stamp: "Formatted at 2026-09-24 17:48:01 on dist-test-slave-r7hf"
server_key: "aba29b0be78a174596ce385d05f568db"
server_key_iv: "628631c2f4d4f67c242dcec513d43096"
server_key_version: "encryptionkey@0"
I20260924 17:48:01.807888 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "3d0b6cda972f47229576f2d82b8fad2d"
format_stamp: "Formatted at 2026-09-24 17:48:01 on dist-test-slave-r7hf"
server_key: "aba29b0be78a174596ce385d05f568db"
server_key_iv: "628631c2f4d4f67c242dcec513d43096"
server_key_version: "encryptionkey@0"
I20260924 17:48:01.809330 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:01.810153 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:01.810369 22630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:48:01.810519 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "3d0b6cda972f47229576f2d82b8fad2d"
format_stamp: "Formatted at 2026-09-24 17:48:01 on dist-test-slave-r7hf"
server_key: "aba29b0be78a174596ce385d05f568db"
server_key_iv: "628631c2f4d4f67c242dcec513d43096"
server_key_version: "encryptionkey@0"
I20260924 17:48:01.810609 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:01.828972 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:01.831307 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:01.831487 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:01.835579 22630 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:45071
I20260924 17:48:01.835620 22697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:45071 every 8 connection(s)
I20260924 17:48:01.836007 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:01.836930 22698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:01.839236 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d: Bootstrap starting.
I20260924 17:48:01.839823 22698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:01.840112 22698 log.cc:824] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:01.840821 22698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d: No bootstrap required, opened a new log
I20260924 17:48:01.840811 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22630
I20260924 17:48:01.840989 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:01.841179 1058 external_mini_cluster.cc:1428] Setting key 8188b121cda03d6fbce412772fdf42f1
I20260924 17:48:01.843359 22698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b6cda972f47229576f2d82b8fad2d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45071 } }
I20260924 17:48:01.843537 22698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:01.843572 22698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0b6cda972f47229576f2d82b8fad2d, State: Initialized, Role: FOLLOWER
I20260924 17:48:01.843724 22698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b6cda972f47229576f2d82b8fad2d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45071 } }
I20260924 17:48:01.843811 22698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:01.843839 22698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:01.843883 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:01.844590 22698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b6cda972f47229576f2d82b8fad2d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45071 } }
I20260924 17:48:01.844695 22698 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d0b6cda972f47229576f2d82b8fad2d; no voters:
I20260924 17:48:01.844852 22698 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:01.844918 22703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:01.845083 22698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:01.845082 22703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0b6cda972f47229576f2d82b8fad2d, State: Running, Role: LEADER
I20260924 17:48:01.845436 22703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b6cda972f47229576f2d82b8fad2d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45071 } }
I20260924 17:48:01.846002 22705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d0b6cda972f47229576f2d82b8fad2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b6cda972f47229576f2d82b8fad2d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45071 } } }
I20260924 17:48:01.846109 22705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:01.846017 22704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0b6cda972f47229576f2d82b8fad2d. Latest consensus state: current_term: 1 leader_uuid: "3d0b6cda972f47229576f2d82b8fad2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d0b6cda972f47229576f2d82b8fad2d" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45071 } } }
I20260924 17:48:01.846150 22704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:01.846302 22710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:01.846853 22710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:01.848573 22710 catalog_manager.cc:1347] Generated new cluster ID: ded2b88c68084d179ac7430686bcd7e4
I20260924 17:48:01.848623 22710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 17:48:01 dist-test-slave-r7hf krb5kdc[22617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:48:01.853869 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:01.841831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60131 (local address 127.1.8.190:45071)
0924 17:48:01.841980 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:01.841989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:01.842022 (+ 33us) server_negotiation.cc:407] Connection header received
0924 17:48:01.842557 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:01.842567 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:01.842631 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:01.842783 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:01.843375 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:01.845064 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:01.846237 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:01.846935 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:01.847047 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:01.850030 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:01.850034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:01.850034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:01.850047 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:01.852085 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:01.852529 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:01.852533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:01.852533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:01.852646 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:01.852883 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:01.852884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:01.852885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:01.853212 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:01.853370 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:01.853505 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 17:48:01.853612 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260924 17:48:01.855075 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:01.861135 22710 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:01.861835 22710 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:01.868208 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:01.858716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33195 (local address 127.1.8.190:45071)
0924 17:48:01.858754 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:01.858756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:01.860777 (+ 2021us) server_negotiation.cc:407] Connection header received
0924 17:48:01.860870 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:01.860871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:01.860882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:01.860947 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:01.861678 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:01.862482 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:01.863442 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:01.863767 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:01.863843 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:01.865247 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:01.865249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:01.865249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:01.865256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:01.866946 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:01.867358 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:01.867359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:01.867359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:01.867428 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:01.867677 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:01.867678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:01.867678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:01.867810 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:01.867916 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:01.868037 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 17:48:01.868111 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:48:01.871852 22710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0b6cda972f47229576f2d82b8fad2d: Generated new TSK 0
I20260924 17:48:01.872078 22710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:01.880487 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22630
W20260924 17:48:01.884789 22615 connection.cc:535] client connection to 127.1.8.190:45071 recv error: Network error: failed to read from TLS socket (remote: 127.1.8.190:45071): Connection reset by peer (error 104)
2026-09-24T17:48:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:01 dist-test-slave-r7hf krb5kdc[22729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:01 dist-test-slave-r7hf krb5kdc[22729](info): set up 2 sockets
Sep 24 17:48:01 dist-test-slave-r7hf krb5kdc[22729](info): commencing operation
krb5kdc: starting...
W20260924 17:48:02.914593 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:02 dist-test-slave-r7hf krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:03.037667 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:38331
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:03.121253 22742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:03.121549 22742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:03.121652 22742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:03.122982 22742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:03.123049 22742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:03.123075 22742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:03.123106 22742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:03.123136 22742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:03.124747 22742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:38331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:03.125039 22742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:03.125276 22742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:03.130198 22750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:03.130267 22742 server_base.cc:1034] running on GCE node
W20260924 17:48:03.130167 22748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:03.130179 22747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:03.130657 22742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:03.130977 22742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:03.132146 22742 hybrid_clock.cc:648] HybridClock initialized: now 1790272083132142 us; error 36 us; skew 500 ppm
Sep 24 17:48:03 dist-test-slave-r7hf krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272083, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:03.134673 22742 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:03.135139 22742 webserver.cc:460] Webserver started at http://127.1.8.190:38441/ using document root <none> and password file <none>
I20260924 17:48:03.135366 22742 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:03.135438 22742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:03.135588 22742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:03.136677 22742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "4c0350bd8b6d4a26ae8524f0d32d195e"
format_stamp: "Formatted at 2026-09-24 17:48:03 on dist-test-slave-r7hf"
server_key: "1d6fde94fdc22b142093e5a5cd82fcaa"
server_key_iv: "2a04549bc26d7cef07b7f872e1962af6"
server_key_version: "encryptionkey@0"
I20260924 17:48:03.137058 22742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "4c0350bd8b6d4a26ae8524f0d32d195e"
format_stamp: "Formatted at 2026-09-24 17:48:03 on dist-test-slave-r7hf"
server_key: "1d6fde94fdc22b142093e5a5cd82fcaa"
server_key_iv: "2a04549bc26d7cef07b7f872e1962af6"
server_key_version: "encryptionkey@0"
I20260924 17:48:03.138506 22742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:03.139256 22757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:03.139421 22742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:48:03.139523 22742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "4c0350bd8b6d4a26ae8524f0d32d195e"
format_stamp: "Formatted at 2026-09-24 17:48:03 on dist-test-slave-r7hf"
server_key: "1d6fde94fdc22b142093e5a5cd82fcaa"
server_key_iv: "2a04549bc26d7cef07b7f872e1962af6"
server_key_version: "encryptionkey@0"
I20260924 17:48:03.139616 22742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:03.155165 22742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:03.157157 22742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:03.157335 22742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:03.161441 22742 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:38331
I20260924 17:48:03.161477 22809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:38331 every 8 connection(s)
I20260924 17:48:03.161886 22742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:03.162928 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22742
I20260924 17:48:03.163043 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:03.163160 22810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:03.163285 1058 external_mini_cluster.cc:1428] Setting key 3745f4bed7e8013e0ab9cf8fe7a8d680
I20260924 17:48:03.166683 22810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e: Bootstrap starting.
I20260924 17:48:03.167312 22810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:03.167678 22810 log.cc:824] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:03.168465 22810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e: No bootstrap required, opened a new log
I20260924 17:48:03.168889 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:03.163834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54415 (local address 127.1.8.190:38331)
0924 17:48:03.164011 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:03.164031 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:03.164063 (+ 32us) server_negotiation.cc:407] Connection header received
0924 17:48:03.164487 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:03.164494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:03.164541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:03.164655 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:03.164891 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:03.165768 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:03.166869 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:03.167291 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:03.167400 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:03.168212 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:03.168216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:03.168217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:03.168391 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:03.168425 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:03.168440 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:03.168534 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:03.168682 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 17:48:03.168713 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 17:48:03.169982 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:03.170948 22810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 38331 } }
I20260924 17:48:03.171150 22810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:03.171200 22810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0350bd8b6d4a26ae8524f0d32d195e, State: Initialized, Role: FOLLOWER
I20260924 17:48:03.171334 22810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 38331 } }
I20260924 17:48:03.171429 22810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:03.171514 22810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:03.171597 22810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:03.172358 22810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 38331 } }
I20260924 17:48:03.172468 22810 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0350bd8b6d4a26ae8524f0d32d195e; no voters:
I20260924 17:48:03.172653 22810 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:03.172705 22820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:03.172848 22820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [term 1 LEADER]: Becoming Leader. State: Replica: 4c0350bd8b6d4a26ae8524f0d32d195e, State: Running, Role: LEADER
I20260924 17:48:03.172910 22810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:03.173050 22820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 38331 } }
I20260924 17:48:03.173394 22820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0350bd8b6d4a26ae8524f0d32d195e. Latest consensus state: current_term: 1 leader_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 38331 } } }
I20260924 17:48:03.173617 22820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:03.173542 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0350bd8b6d4a26ae8524f0d32d195e" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 38331 } } }
I20260924 17:48:03.173892 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:03.174248 22829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:03.174863 22829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:03.175067 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:03.171434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59359 (local address 127.1.8.190:38331)
0924 17:48:03.171479 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:03.171481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:03.171598 (+ 117us) server_negotiation.cc:407] Connection header received
0924 17:48:03.171680 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:03.171682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:03.171693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:03.171754 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:03.171850 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:03.172541 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:03.173639 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:03.173967 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:03.174018 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:03.174811 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:03.174813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:03.174813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:03.174818 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:03.174832 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:03.174836 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:03.174889 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:03.174993 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 17:48:03.175005 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:48:03.176391 22829 catalog_manager.cc:1347] Generated new cluster ID: 121363aa5f2047788ab1994df8c0df51
I20260924 17:48:03.176445 22829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:03.182751 22829 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:03.183306 22829 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:03.188632 22829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0350bd8b6d4a26ae8524f0d32d195e: Generated new TSK 0
I20260924 17:48:03.188820 22829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:03.191107 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22742
W20260924 17:48:03.195652 22727 connection.cc:535] client connection to 127.1.8.190:38331 recv error: Network error: failed to read from TLS socket (remote: 127.1.8.190:38331): Connection reset by peer (error 104)
2026-09-24T17:48:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:03 dist-test-slave-r7hf krb5kdc[22841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:03 dist-test-slave-r7hf krb5kdc[22841](info): set up 2 sockets
Sep 24 17:48:03 dist-test-slave-r7hf krb5kdc[22841](info): commencing operation
krb5kdc: starting...
W20260924 17:48:04.222677 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:04 dist-test-slave-r7hf krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:04.345594 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44823
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:04.428654 22854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:04.428922 22854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:04.428958 22854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:04.430154 22854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:04.430235 22854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:04.430262 22854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:04.430294 22854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:04.430341 22854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:04.431887 22854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:04.432190 22854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:04.432428 22854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:04.437736 22859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:04.437758 22862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:04.437827 22854 server_base.cc:1034] running on GCE node
W20260924 17:48:04.437954 22860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:04.438216 22854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:04.438575 22854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:04.439738 22854 hybrid_clock.cc:648] HybridClock initialized: now 1790272084439731 us; error 43 us; skew 500 ppm
Sep 24 17:48:04 dist-test-slave-r7hf krb5kdc[22841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272084, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:04.442526 22854 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:04.442935 22854 webserver.cc:460] Webserver started at http://127.1.8.190:43017/ using document root <none> and password file <none>
I20260924 17:48:04.443126 22854 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:04.443168 22854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:04.443257 22854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:04.444293 22854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "0f8a2a26c20341ddbd27b37124aa1017"
format_stamp: "Formatted at 2026-09-24 17:48:04 on dist-test-slave-r7hf"
server_key: "39656609f1e2e76e3bb1ec5eb614c214"
server_key_iv: "f1ffa5c55817a514617b75df8e9c682f"
server_key_version: "encryptionkey@0"
I20260924 17:48:04.444609 22854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "0f8a2a26c20341ddbd27b37124aa1017"
format_stamp: "Formatted at 2026-09-24 17:48:04 on dist-test-slave-r7hf"
server_key: "39656609f1e2e76e3bb1ec5eb614c214"
server_key_iv: "f1ffa5c55817a514617b75df8e9c682f"
server_key_version: "encryptionkey@0"
I20260924 17:48:04.445920 22854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:48:04.446785 22869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:04.447010 22854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:48:04.447093 22854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "0f8a2a26c20341ddbd27b37124aa1017"
format_stamp: "Formatted at 2026-09-24 17:48:04 on dist-test-slave-r7hf"
server_key: "39656609f1e2e76e3bb1ec5eb614c214"
server_key_iv: "f1ffa5c55817a514617b75df8e9c682f"
server_key_version: "encryptionkey@0"
I20260924 17:48:04.447149 22854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:04.450486 22854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:04.452574 22854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:04.452685 22854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:04.456462 22854 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:44823
I20260924 17:48:04.456511 22921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:44823 every 8 connection(s)
I20260924 17:48:04.456837 22854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:04.457849 22922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:04.460333 22922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017: Bootstrap starting.
I20260924 17:48:04.460582 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22854
I20260924 17:48:04.460692 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:04.460887 1058 external_mini_cluster.cc:1428] Setting key 134f4c23dbc8cd44119bc6749c3ee83e
I20260924 17:48:04.461057 22922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:04.461419 22922 log.cc:824] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:04.462214 22922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017: No bootstrap required, opened a new log
I20260924 17:48:04.463654 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:04.461467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46501 (local address 127.1.8.190:44823)
0924 17:48:04.461735 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:04.461745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:04.461811 (+ 66us) server_negotiation.cc:407] Connection header received
0924 17:48:04.462289 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:04.462296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:04.462366 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:04.462556 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:04.463199 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:04.463203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:04.463203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:04.463207 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:04.463245 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:04.463256 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:04.463324 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:04.463399 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 17:48:04.463454 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":156,"threads_started":1}
I20260924 17:48:04.464634 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:04.464768 22922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8a2a26c20341ddbd27b37124aa1017" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44823 } }
I20260924 17:48:04.464932 22922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:04.464963 22922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8a2a26c20341ddbd27b37124aa1017, State: Initialized, Role: FOLLOWER
I20260924 17:48:04.465095 22922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8a2a26c20341ddbd27b37124aa1017" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44823 } }
I20260924 17:48:04.465190 22922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:04.465222 22922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:04.465271 22922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:04.466075 22922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8a2a26c20341ddbd27b37124aa1017" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44823 } }
I20260924 17:48:04.466202 22922 leader_election.cc:302] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f8a2a26c20341ddbd27b37124aa1017; no voters:
I20260924 17:48:04.466377 22922 leader_election.cc:288] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:04.466450 22932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:04.466653 22922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:04.466637 22932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [term 1 LEADER]: Becoming Leader. State: Replica: 0f8a2a26c20341ddbd27b37124aa1017, State: Running, Role: LEADER
I20260924 17:48:04.466843 22932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8a2a26c20341ddbd27b37124aa1017" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44823 } }
I20260924 17:48:04.467307 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f8a2a26c20341ddbd27b37124aa1017. Latest consensus state: current_term: 1 leader_uuid: "0f8a2a26c20341ddbd27b37124aa1017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8a2a26c20341ddbd27b37124aa1017" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44823 } } }
I20260924 17:48:04.467396 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:04.467492 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:04.466223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44351 (local address 127.1.8.190:44823)
0924 17:48:04.466269 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:04.466271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:04.466435 (+ 164us) server_negotiation.cc:407] Connection header received
0924 17:48:04.466542 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:04.466543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:04.466548 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:04.466580 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:04.467246 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:04.467247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:04.467247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:04.467251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:04.467266 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:04.467270 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:04.467299 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:04.467410 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 17:48:04.467424 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:48:04.467708 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f8a2a26c20341ddbd27b37124aa1017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8a2a26c20341ddbd27b37124aa1017" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44823 } } }
I20260924 17:48:04.467839 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017 [sys.catalog]: This master's current role is: LEADER
W20260924 17:48:04.468465 22948 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:04.468532 22948 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:48:04.468605 22941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:04.468808 22941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:04.470214 22941 catalog_manager.cc:1347] Generated new cluster ID: 7374e30b22fa42d58adf52029e0878b5
I20260924 17:48:04.470264 22941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:04.476862 22941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:04.477061 22941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:04.486027 22941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f8a2a26c20341ddbd27b37124aa1017: Generated new TSK 0
I20260924 17:48:04.486265 22941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:04.493151 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22854
2026-09-24T17:48:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:04 dist-test-slave-r7hf krb5kdc[22953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:04 dist-test-slave-r7hf krb5kdc[22953](info): set up 2 sockets
Sep 24 17:48:04 dist-test-slave-r7hf krb5kdc[22953](info): commencing operation
krb5kdc: starting...
W20260924 17:48:05.550594 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:05 dist-test-slave-r7hf krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:05.678061 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:46021
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:35001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:46021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:05.761018 22966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:05.761232 22966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:05.761256 22966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:05.762617 22966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:05.762661 22966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:05.762676 22966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:05.762688 22966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:05.762701 22966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:05.764247 22966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:35001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:46021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:46021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:05.764514 22966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:05.764750 22966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:05.769688 22972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:05.769688 22971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:05.769688 22974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:05.769887 22966 server_base.cc:1034] running on GCE node
I20260924 17:48:05.770164 22966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:05.770527 22966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:05.771692 22966 hybrid_clock.cc:648] HybridClock initialized: now 1790272085771678 us; error 27 us; skew 500 ppm
Sep 24 17:48:05 dist-test-slave-r7hf krb5kdc[22953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272085, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:05.774245 22966 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:05.774823 22966 webserver.cc:460] Webserver started at http://127.1.8.190:46101/ using document root <none> and password file <none>
I20260924 17:48:05.775082 22966 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:05.775151 22966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:05.775300 22966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:05.776396 22966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "2f04d7a3eaaa45028493015903cad5ad"
format_stamp: "Formatted at 2026-09-24 17:48:05 on dist-test-slave-r7hf"
server_key: "8611ff4f3887dded86368dc2f510a40a"
server_key_iv: "058cbe9db3e7b8a85652aacc4c2312da"
server_key_version: "encryptionkey@0"
I20260924 17:48:05.776767 22966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "2f04d7a3eaaa45028493015903cad5ad"
format_stamp: "Formatted at 2026-09-24 17:48:05 on dist-test-slave-r7hf"
server_key: "8611ff4f3887dded86368dc2f510a40a"
server_key_iv: "058cbe9db3e7b8a85652aacc4c2312da"
server_key_version: "encryptionkey@0"
I20260924 17:48:05.778110 22966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:05.778918 22981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:05.779096 22966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:05.779188 22966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "2f04d7a3eaaa45028493015903cad5ad"
format_stamp: "Formatted at 2026-09-24 17:48:05 on dist-test-slave-r7hf"
server_key: "8611ff4f3887dded86368dc2f510a40a"
server_key_iv: "058cbe9db3e7b8a85652aacc4c2312da"
server_key_version: "encryptionkey@0"
I20260924 17:48:05.779273 22966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:05.788934 22966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:05.790897 22966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:05.791062 22966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:05.795660 22966 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:46021
I20260924 17:48:05.795696 23033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:46021 every 8 connection(s)
I20260924 17:48:05.796068 22966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:05.796944 23034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:05.799369 23034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad: Bootstrap starting.
I20260924 17:48:05.799983 23034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:05.800280 23034 log.cc:824] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:05.801010 23034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad: No bootstrap required, opened a new log
I20260924 17:48:05.802731 23034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f04d7a3eaaa45028493015903cad5ad" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46021 } }
I20260924 17:48:05.802871 23034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:05.802893 23034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f04d7a3eaaa45028493015903cad5ad, State: Initialized, Role: FOLLOWER
I20260924 17:48:05.803068 23034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f04d7a3eaaa45028493015903cad5ad" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46021 } }
I20260924 17:48:05.803196 23034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:05.803236 23034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:05.803290 23034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:05.803284 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 22966
I20260924 17:48:05.803380 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:05.803575 1058 external_mini_cluster.cc:1428] Setting key ac3bd56512adf7c7ac1ca7e8df3a8e20
I20260924 17:48:05.804423 23034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f04d7a3eaaa45028493015903cad5ad" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46021 } }
I20260924 17:48:05.804515 23034 leader_election.cc:302] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f04d7a3eaaa45028493015903cad5ad; no voters:
I20260924 17:48:05.804735 23034 leader_election.cc:288] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:05.804759 23039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:05.804914 23039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [term 1 LEADER]: Becoming Leader. State: Replica: 2f04d7a3eaaa45028493015903cad5ad, State: Running, Role: LEADER
I20260924 17:48:05.805056 23034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:05.805037 23039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f04d7a3eaaa45028493015903cad5ad" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46021 } }
I20260924 17:48:05.805486 23041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f04d7a3eaaa45028493015903cad5ad. Latest consensus state: current_term: 1 leader_uuid: "2f04d7a3eaaa45028493015903cad5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f04d7a3eaaa45028493015903cad5ad" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46021 } } }
I20260924 17:48:05.805601 23041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:05.806167 23044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:05.806668 23044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:05.808326 23044 catalog_manager.cc:1347] Generated new cluster ID: d418faaeaf50431cb107909a322b5cfa
I20260924 17:48:05.808370 23044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:05.808905 23040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f04d7a3eaaa45028493015903cad5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f04d7a3eaaa45028493015903cad5ad" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 46021 } } }
I20260924 17:48:05.808980 23040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad [sys.catalog]: This master's current role is: LEADER
Sep 24 17:48:05 dist-test-slave-r7hf krb5kdc[22953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:48:05.816103 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:05.804115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48271 (local address 127.1.8.190:46021)
0924 17:48:05.804322 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:05.804331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:05.804366 (+ 35us) server_negotiation.cc:407] Connection header received
0924 17:48:05.804462 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:05.804467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:05.804543 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:05.804697 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:05.805451 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:05.806910 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:05.808000 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:05.808557 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:05.808656 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:05.811721 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:05.811725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:05.811725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:05.811739 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:05.814159 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:05.814639 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:05.814643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:05.814643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:05.814774 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:05.815005 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:05.815006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:05.815006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:05.815338 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:05.815498 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:05.815635 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 17:48:05.815872 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260924 17:48:05.817165 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:05.825479 23044 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:05.826207 23044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:05.826380 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:05.818896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48483 (local address 127.1.8.190:46021)
0924 17:48:05.818940 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:05.818942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:05.819049 (+ 107us) server_negotiation.cc:407] Connection header received
0924 17:48:05.819082 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:05.819083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:05.819094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:05.819163 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:05.819750 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:05.820540 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:05.821490 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:05.821847 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:05.821922 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:05.823396 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:05.823397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:05.823397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:05.823405 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:05.825129 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:05.825559 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:05.825560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:05.825560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:05.825624 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:05.825841 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:05.825842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:05.825842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:05.825973 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:05.826084 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:05.826194 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 17:48:05.826288 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:48:05.833817 23044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f04d7a3eaaa45028493015903cad5ad: Generated new TSK 0
I20260924 17:48:05.834051 23044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:05.837910 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 22966
2026-09-24T17:48:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:05 dist-test-slave-r7hf krb5kdc[23065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:05 dist-test-slave-r7hf krb5kdc[23065](info): set up 2 sockets
Sep 24 17:48:05 dist-test-slave-r7hf krb5kdc[23065](info): commencing operation
krb5kdc: starting...
W20260924 17:48:06.882606 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:06 dist-test-slave-r7hf krb5kdc[23065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:07.010879 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39571
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:39571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:07.094617 23078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:07.094909 23078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:07.094954 23078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:07.096169 23078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:07.096252 23078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:07.096279 23078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:07.096307 23078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:07.096340 23078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:07.097965 23078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:39571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:39571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:07.098266 23078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:07.098562 23078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:48:07.103941 23078 server_base.cc:1034] running on GCE node
W20260924 17:48:07.103890 23083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:07.103969 23086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:07.103899 23084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:07.104493 23078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:07.104787 23078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:07.105963 23078 hybrid_clock.cc:648] HybridClock initialized: now 1790272087105948 us; error 36 us; skew 500 ppm
Sep 24 17:48:07 dist-test-slave-r7hf krb5kdc[23065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:07.114101 23078 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:07.114686 23078 webserver.cc:460] Webserver started at http://127.1.8.190:44249/ using document root <none> and password file <none>
I20260924 17:48:07.114903 23078 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:07.114948 23078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:07.115046 23078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:07.116102 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "bf1fdabb74ee41238f3383d004fa00a6"
format_stamp: "Formatted at 2026-09-24 17:48:07 on dist-test-slave-r7hf"
server_key: "e2129b566247b4d72f5705aefec2a552"
server_key_iv: "2a41f3d495ab9ec4e9a497cd8a1c64f2"
server_key_version: "encryptionkey@0"
I20260924 17:48:07.116411 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "bf1fdabb74ee41238f3383d004fa00a6"
format_stamp: "Formatted at 2026-09-24 17:48:07 on dist-test-slave-r7hf"
server_key: "e2129b566247b4d72f5705aefec2a552"
server_key_iv: "2a41f3d495ab9ec4e9a497cd8a1c64f2"
server_key_version: "encryptionkey@0"
I20260924 17:48:07.117741 23078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:07.118500 23093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:07.118700 23078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:48:07.118814 23078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "bf1fdabb74ee41238f3383d004fa00a6"
format_stamp: "Formatted at 2026-09-24 17:48:07 on dist-test-slave-r7hf"
server_key: "e2129b566247b4d72f5705aefec2a552"
server_key_iv: "2a41f3d495ab9ec4e9a497cd8a1c64f2"
server_key_version: "encryptionkey@0"
I20260924 17:48:07.118903 23078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:07.144603 23078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:07.146888 23078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:07.147070 23078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:07.150967 23078 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:39571
I20260924 17:48:07.151000 23145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:39571 every 8 connection(s)
I20260924 17:48:07.151364 23078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:07.152277 23146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:07.154703 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6: Bootstrap starting.
I20260924 17:48:07.155301 23146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:07.155588 23146 log.cc:824] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:07.156397 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6: No bootstrap required, opened a new log
I20260924 17:48:07.156528 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23078
I20260924 17:48:07.156637 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:07.156823 1058 external_mini_cluster.cc:1428] Setting key c838b17c486d9efd057d2f84d4e88f78
I20260924 17:48:07.158535 23146 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1fdabb74ee41238f3383d004fa00a6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39571 } }
I20260924 17:48:07.158699 23146 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:07.158735 23146 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1fdabb74ee41238f3383d004fa00a6, State: Initialized, Role: FOLLOWER
I20260924 17:48:07.158931 23146 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1fdabb74ee41238f3383d004fa00a6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39571 } }
I20260924 17:48:07.159026 23146 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:07.159075 23146 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:07.159125 23146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:07.159696 23146 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1fdabb74ee41238f3383d004fa00a6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39571 } }
I20260924 17:48:07.159789 23146 leader_election.cc:302] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1fdabb74ee41238f3383d004fa00a6; no voters:
I20260924 17:48:07.160049 23146 leader_election.cc:288] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:07.160072 23151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:07.160184 23151 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [term 1 LEADER]: Becoming Leader. State: Replica: bf1fdabb74ee41238f3383d004fa00a6, State: Running, Role: LEADER
I20260924 17:48:07.160323 23146 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:07.160326 23151 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1fdabb74ee41238f3383d004fa00a6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39571 } }
I20260924 17:48:07.160786 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1fdabb74ee41238f3383d004fa00a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1fdabb74ee41238f3383d004fa00a6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39571 } } }
I20260924 17:48:07.161036 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:07.161451 23160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:07.160959 23152 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1fdabb74ee41238f3383d004fa00a6. Latest consensus state: current_term: 1 leader_uuid: "bf1fdabb74ee41238f3383d004fa00a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1fdabb74ee41238f3383d004fa00a6" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 39571 } } }
I20260924 17:48:07.161975 23152 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:07.162006 23160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:07.163889 23160 catalog_manager.cc:1347] Generated new cluster ID: 73022239170c41698e4b2a19b8a146f2
I20260924 17:48:07.163913 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:07.157342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48017 (local address 127.1.8.190:39571)
0924 17:48:07.157546 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:07.157553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:07.157579 (+ 26us) server_negotiation.cc:407] Connection header received
0924 17:48:07.158519 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:07.158526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:07.158577 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:07.158716 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:07.158959 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:07.160452 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:07.161631 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:07.162260 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:07.162366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:07.163188 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:07.163191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:07.163192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:07.163420 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:07.163459 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:07.163488 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:07.163556 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:07.163637 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 17:48:07.163705 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260924 17:48:07.163942 23160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:07.167213 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:07.169528 23160 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:07.170219 23160 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:07.172456 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:07.168967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53931 (local address 127.1.8.190:39571)
0924 17:48:07.169015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:07.169017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:07.169365 (+ 348us) server_negotiation.cc:407] Connection header received
0924 17:48:07.169567 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:07.169568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:07.169579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:07.169631 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:07.169837 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:07.170351 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:07.171139 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:07.171348 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:07.171429 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:07.172175 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:07.172176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:07.172176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:07.172181 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:07.172197 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:07.172201 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:07.172255 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:07.172313 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 17:48:07.172355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 17:48:07.176347 23160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1fdabb74ee41238f3383d004fa00a6: Generated new TSK 0
I20260924 17:48:07.176539 23160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:07.183805 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23078
2026-09-24T17:48:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:07 dist-test-slave-r7hf krb5kdc[23177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:07 dist-test-slave-r7hf krb5kdc[23177](info): set up 2 sockets
Sep 24 17:48:07 dist-test-slave-r7hf krb5kdc[23177](info): commencing operation
krb5kdc: starting...
W20260924 17:48:08.222662 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:08 dist-test-slave-r7hf krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:08.350051 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45533
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:08.433786 23190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:08.434090 23190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:08.434165 23190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:08.435707 23190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:08.435782 23190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:08.435809 23190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:08.435842 23190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:08.435868 23190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:08.437398 23190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:08.437695 23190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:08.437934 23190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:08.443063 23198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:08.443051 23195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:08.443241 23190 server_base.cc:1034] running on GCE node
W20260924 17:48:08.443132 23196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:08.443537 23190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:08.443837 23190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:08.445129 23190 hybrid_clock.cc:648] HybridClock initialized: now 1790272088445103 us; error 34 us; skew 500 ppm
Sep 24 17:48:08 dist-test-slave-r7hf krb5kdc[23177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272088, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:08.447635 23190 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:08.448062 23190 webserver.cc:460] Webserver started at http://127.1.8.190:45253/ using document root <none> and password file <none>
I20260924 17:48:08.448261 23190 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:08.448304 23190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:08.448395 23190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:08.449414 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "b7861f2c744c4015876c71f372f3a840"
format_stamp: "Formatted at 2026-09-24 17:48:08 on dist-test-slave-r7hf"
server_key: "4d86dd78c1bf9f974eeb41422ba169eb"
server_key_iv: "c5c7d2dae33928966c8ee79b8a8677ab"
server_key_version: "encryptionkey@0"
I20260924 17:48:08.449714 23190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "b7861f2c744c4015876c71f372f3a840"
format_stamp: "Formatted at 2026-09-24 17:48:08 on dist-test-slave-r7hf"
server_key: "4d86dd78c1bf9f974eeb41422ba169eb"
server_key_iv: "c5c7d2dae33928966c8ee79b8a8677ab"
server_key_version: "encryptionkey@0"
I20260924 17:48:08.451100 23190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:48:08.451792 23205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:08.451964 23190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:48:08.452050 23190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "b7861f2c744c4015876c71f372f3a840"
format_stamp: "Formatted at 2026-09-24 17:48:08 on dist-test-slave-r7hf"
server_key: "4d86dd78c1bf9f974eeb41422ba169eb"
server_key_iv: "c5c7d2dae33928966c8ee79b8a8677ab"
server_key_version: "encryptionkey@0"
I20260924 17:48:08.452107 23190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:08.455397 23190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:08.457291 23190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:08.457398 23190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:08.461016 23190 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:45533
I20260924 17:48:08.461071 23257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:45533 every 8 connection(s)
I20260924 17:48:08.461390 23190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:08.462481 23258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:08.464789 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840: Bootstrap starting.
I20260924 17:48:08.464879 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23190
I20260924 17:48:08.464964 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:08.465157 1058 external_mini_cluster.cc:1428] Setting key 67acf752eb95b5bd64c16b68018b43c1
I20260924 17:48:08.465535 23258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:08.466121 23258 log.cc:824] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:08.466981 23258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840: No bootstrap required, opened a new log
I20260924 17:48:08.467885 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:08.465711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58243 (local address 127.1.8.190:45533)
0924 17:48:08.465958 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:08.465966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:08.465994 (+ 28us) server_negotiation.cc:407] Connection header received
0924 17:48:08.466555 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:08.466563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:08.466591 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:08.466715 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:08.467421 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:08.467424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:08.467424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:08.467428 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:08.467465 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:08.467475 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:08.467545 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:08.467620 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 17:48:08.467673 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
I20260924 17:48:08.468885 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:08.469508 23258 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7861f2c744c4015876c71f372f3a840" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45533 } }
I20260924 17:48:08.469673 23258 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 17:48:08.469719 23258 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7861f2c744c4015876c71f372f3a840, State: Initialized, Role: FOLLOWER
I20260924 17:48:08.469882 23258 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7861f2c744c4015876c71f372f3a840" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45533 } }
I20260924 17:48:08.469982 23258 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 17:48:08.470037 23258 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 17:48:08.470088 23258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:08.470861 23258 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7861f2c744c4015876c71f372f3a840" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45533 } }
I20260924 17:48:08.471015 23258 leader_election.cc:302] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7861f2c744c4015876c71f372f3a840; no voters:
I20260924 17:48:08.471201 23258 leader_election.cc:288] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 17:48:08.471248 23268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:08.471407 23268 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [term 1 LEADER]: Becoming Leader. State: Replica: b7861f2c744c4015876c71f372f3a840, State: Running, Role: LEADER
I20260924 17:48:08.471455 23258 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:08.471558 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:08.470266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49977 (local address 127.1.8.190:45533)
0924 17:48:08.470348 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:08.470349 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:08.470359 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:48:08.470436 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:08.470437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:08.470442 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:08.470503 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:08.471099 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:08.471101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:08.471101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 17:48:08.471105 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:08.471122 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 17:48:08.471126 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:08.471175 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:08.471410 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 17:48:08.471423 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 17:48:08.471562 23268 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7861f2c744c4015876c71f372f3a840" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45533 } }
I20260924 17:48:08.472003 23268 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7861f2c744c4015876c71f372f3a840. Latest consensus state: current_term: 1 leader_uuid: "b7861f2c744c4015876c71f372f3a840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7861f2c744c4015876c71f372f3a840" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45533 } } }
I20260924 17:48:08.472136 23268 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:08.472013 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7861f2c744c4015876c71f372f3a840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7861f2c744c4015876c71f372f3a840" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45533 } } }
I20260924 17:48:08.472282 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:08.472394 23277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:08.472935 23277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:08.474648 23277 catalog_manager.cc:1347] Generated new cluster ID: 2940ef02130941c7a30bf51177924dbf
I20260924 17:48:08.474705 23277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:08.482460 23277 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:08.482657 23277 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:08.487775 23277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7861f2c744c4015876c71f372f3a840: Generated new TSK 0
I20260924 17:48:08.488032 23277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:08.493106 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23190
2026-09-24T17:48:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1309 ms)
[----------] 18 tests from AuthTokenIssuingTest (24115 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:08 dist-test-slave-r7hf krb5kdc[23289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:08 dist-test-slave-r7hf krb5kdc[23289](info): set up 2 sockets
Sep 24 17:48:08 dist-test-slave-r7hf krb5kdc[23289](info): commencing operation
krb5kdc: starting...
W20260924 17:48:09.538622 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:09 dist-test-slave-r7hf krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:09.666949 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44845
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44845,127.1.8.189:37613,127.1.8.188:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:09.749656 23302 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:48:09.749912 23302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:09.749951 23302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:09.749994 23302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:09.751192 23302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:09.751257 23302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:09.751282 23302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:09.751304 23302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:09.751330 23302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:09.752827 23302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44845,127.1.8.189:37613,127.1.8.188:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:44845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:09.753159 23302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:09.753391 23302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:48:09.758500 23302 server_base.cc:1034] running on GCE node
W20260924 17:48:09.758351 23307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:09.758364 23308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:09.758352 23310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:09.758931 23302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:09.759296 23302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:09.760474 23302 hybrid_clock.cc:648] HybridClock initialized: now 1790272089760462 us; error 50 us; skew 500 ppm
Sep 24 17:48:09 dist-test-slave-r7hf krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:09.763033 23302 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:09.763513 23302 webserver.cc:460] Webserver started at http://127.1.8.190:43171/ using document root <none> and password file <none>
I20260924 17:48:09.763756 23302 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:09.763841 23302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:09.763989 23302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:09.765093 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "7de764c3a1764302a5f12a159bd3a9a9"
format_stamp: "Formatted at 2026-09-24 17:48:09 on dist-test-slave-r7hf"
server_key: "cbfca2a5500bcfc30ad7b856af4ad023"
server_key_iv: "c2cdb2ba5d4053e63fd7de9520782f46"
server_key_version: "encryptionkey@0"
I20260924 17:48:09.765445 23302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "7de764c3a1764302a5f12a159bd3a9a9"
format_stamp: "Formatted at 2026-09-24 17:48:09 on dist-test-slave-r7hf"
server_key: "cbfca2a5500bcfc30ad7b856af4ad023"
server_key_iv: "c2cdb2ba5d4053e63fd7de9520782f46"
server_key_version: "encryptionkey@0"
I20260924 17:48:09.766853 23302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:09.767643 23317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:09.767899 23302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 17:48:09.768033 23302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "7de764c3a1764302a5f12a159bd3a9a9"
format_stamp: "Formatted at 2026-09-24 17:48:09 on dist-test-slave-r7hf"
server_key: "cbfca2a5500bcfc30ad7b856af4ad023"
server_key_iv: "c2cdb2ba5d4053e63fd7de9520782f46"
server_key_version: "encryptionkey@0"
I20260924 17:48:09.768108 23302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:09.781677 23302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:09.784020 23302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:09.784191 23302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:09.788626 23302 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:44845
I20260924 17:48:09.788681 23369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:44845 every 8 connection(s)
I20260924 17:48:09.789062 23302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:09.790181 23370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:09.792304 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23302
I20260924 17:48:09.792431 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:09.792320 23370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:09.792624 1058 external_mini_cluster.cc:1428] Setting key e1d6888f7a21e5e920fd927c8560fa09
Sep 24 17:48:09 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.189@KRBTEST.COM" created.
I20260924 17:48:09.803319 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.793140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44845 (local address 127.0.0.1:56742)
0924 17:48:09.793325 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.793370 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:09.793910 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:09.794230 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:09.794233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:09.794597 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:09.794762 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:09.794767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.795786 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.795787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:09.796479 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:09.796481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.797036 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.797037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:09.797104 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:09.797740 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:09.797749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:09.799481 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:09.801470 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:09.801477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:09.801479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:09.801700 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:09.802043 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:09.802044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:09.802044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:09.802135 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:09.802629 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:09.802637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:09.802723 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 17:48:09.802830 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 17:48:09.802985 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260924 17:48:09.803330 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.793171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.1.8.190:44845)
0924 17:48:09.793370 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:09.793380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:09.793486 (+ 106us) server_negotiation.cc:407] Connection header received
0924 17:48:09.794102 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:09.794104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:09.794132 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:09.794225 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:09.794865 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.795667 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.796584 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.796916 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.797056 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:09.799593 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:09.799601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:09.799601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:09.799608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:09.801358 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:09.801862 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:09.801865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:09.801866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:09.801955 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:09.802238 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:09.802239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:09.802239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:09.802548 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:09.802625 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:09.802885 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 17:48:09.803036 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":171,"thread_start_us":105,"threads_started":1}
I20260924 17:48:09.804102 23370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:09.804452 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.804342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:48954)
0924 17:48:09.804377 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.804422 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 17:48:09.804575 23321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
I20260924 17:48:09.805070 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.804990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:48962)
0924 17:48:09.805026 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.805050 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 17:48:09.805188 23370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:37613: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab.
Entry for principal kudu/127.1.8.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab.
I20260924 17:48:09.821604 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:37613
--webserver_interface=127.1.8.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab
--principal=kudu/127.1.8.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44845,127.1.8.189:37613,127.1.8.188:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:48:09.854472 23370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } attempt: 1
I20260924 17:48:09.854991 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.854843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:48978)
0924 17:48:09.854911 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.854951 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 17:48:09.855329 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.855221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:48986)
0924 17:48:09.855268 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.855312 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 17:48:09.855463 23370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:37613: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
W20260924 17:48:09.905422 23377 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:48:09.905669 23377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:09.905687 23377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:09.905709 23377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:09.907056 23377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:09.907102 23377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:09.907116 23377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:09.907172 23377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:09.907204 23377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:09.908733 23377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44845,127.1.8.189:37613,127.1.8.188:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab
--principal=kudu/127.1.8.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.8.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:09.909008 23377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:09.909243 23377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 17:48:09.914216 23377 server_base.cc:1034] running on GCE node
W20260924 17:48:09.914196 23383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:09.914201 23385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:09.914181 23382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:09.914747 23377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:09.915050 23377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:09.916194 23377 hybrid_clock.cc:648] HybridClock initialized: now 1790272089916175 us; error 40 us; skew 500 ppm
Sep 24 17:48:09 dist-test-slave-r7hf krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:09.918814 23377 init.cc:377] Logged in from keytab as kudu/127.1.8.189@KRBTEST.COM (short username kudu)
I20260924 17:48:09.919210 23377 webserver.cc:460] Webserver started at http://127.1.8.189:35921/ using document root <none> and password file <none>
I20260924 17:48:09.919399 23377 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:09.919448 23377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:09.919538 23377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:09.920574 23377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data/instance:
uuid: "fd3793cdca1b4accbec26807d6650574"
format_stamp: "Formatted at 2026-09-24 17:48:09 on dist-test-slave-r7hf"
server_key: "a2b21afe4c37e11f4372c9554331a34f"
server_key_iv: "f3ae08ec428b9fbe8616433795fd3f85"
server_key_version: "encryptionkey@0"
I20260924 17:48:09.920872 23377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal/instance:
uuid: "fd3793cdca1b4accbec26807d6650574"
format_stamp: "Formatted at 2026-09-24 17:48:09 on dist-test-slave-r7hf"
server_key: "a2b21afe4c37e11f4372c9554331a34f"
server_key_iv: "f3ae08ec428b9fbe8616433795fd3f85"
server_key_version: "encryptionkey@0"
I20260924 17:48:09.922163 23377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:09.922940 23392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:09.923116 23377 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:09.923220 23377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal
uuid: "fd3793cdca1b4accbec26807d6650574"
format_stamp: "Formatted at 2026-09-24 17:48:09 on dist-test-slave-r7hf"
server_key: "a2b21afe4c37e11f4372c9554331a34f"
server_key_iv: "f3ae08ec428b9fbe8616433795fd3f85"
server_key_version: "encryptionkey@0"
I20260924 17:48:09.923295 23377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:09.923612 23370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } attempt: 2
I20260924 17:48:09.924006 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.923883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:49000)
0924 17:48:09.923941 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.923975 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:48:09.924324 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.924233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:49010)
0924 17:48:09.924278 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.924301 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:48:09.924417 23370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:37613: Network error: Client connection negotiation failed: client connection to 127.1.8.189:37613: connect: Connection refused (error 111)
I20260924 17:48:09.948179 23377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:09.950848 23377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:09.951027 23377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:09.954994 23377 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.189:37613
I20260924 17:48:09.955060 23444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.189:37613 every 8 connection(s)
I20260924 17:48:09.955427 23377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
I20260924 17:48:09.956435 23445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:09.957073 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23377
I20260924 17:48:09.957158 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-1/wal/instance
I20260924 17:48:09.957351 1058 external_mini_cluster.cc:1428] Setting key 889830d4661dcb356958e37f691b8965
I20260924 17:48:09.958876 23445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.8.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.188@KRBTEST.COM" created.
Sep 24 17:48:09 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:48:09.972083 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.959694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.1.8.190:44845)
0924 17:48:09.959767 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:09.959770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:09.960003 (+ 233us) server_negotiation.cc:407] Connection header received
0924 17:48:09.960522 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:09.960523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:09.960538 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:09.960595 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:09.961312 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.961834 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.962941 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.963170 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.963262 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:09.968524 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:09.968526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:09.968526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:09.968536 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:09.970325 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:09.970872 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:09.970873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:09.970874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:09.970946 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:09.971208 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:09.971209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:09.971210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:09.971386 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:09.971465 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:09.971856 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 17:48:09.971959 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 17:48:09.972182 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.959675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44845 (local address 127.0.0.1:56750)
0924 17:48:09.959859 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.959901 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:09.960412 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:09.960676 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:09.960680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:09.961014 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:09.961213 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:09.961217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.961939 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.961941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:09.962838 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:09.962840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.963283 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.963284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:09.963414 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:09.964013 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:09.964022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:09.968389 (+ 4367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:09.970475 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:09.970488 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:09.970492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:09.970784 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:09.971030 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:09.971030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:09.971031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:09.971139 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:09.971457 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:09.971461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:09.971667 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 17:48:09.971763 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 17:48:09.971961 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260924 17:48:09.972955 23445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:48:09 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:09.982534 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.973286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:49018)
0924 17:48:09.973350 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.973364 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:09.973416 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:09.973621 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:09.973622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:09.973791 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:09.973861 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:09.973863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.974800 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.974801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:09.975702 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:09.975705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.976221 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.976222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:09.976313 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:09.976820 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:09.976824 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:09.979159 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:09.981342 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:09.981343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:09.981344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:09.981530 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:09.981730 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:09.981731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:09.981731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:09.981783 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:09.982098 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:09.982098 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:09.982146 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 17:48:09.982211 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 17:48:09.982433 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":84}
I20260924 17:48:09.982546 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.973326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.1.8.189:37613)
0924 17:48:09.973485 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:09.973489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:09.973504 (+ 15us) server_negotiation.cc:407] Connection header received
0924 17:48:09.973523 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:09.973525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:09.973536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:09.973597 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:09.973941 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.974717 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.975797 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:09.976131 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:09.976196 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:09.979319 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:09.979331 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:09.979332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:09.979339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:09.981236 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:09.981617 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:09.981620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:09.981621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:09.981680 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:09.981856 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:09.981856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:09.981857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:09.982026 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:09.982121 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:09.982239 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 17:48:09.982344 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 17:48:09.983328 23445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:09.983708 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.983595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38556)
0924 17:48:09.983642 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.983689 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:48:09.983855 23396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
I20260924 17:48:09.984344 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:09.984220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38558)
0924 17:48:09.984291 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:09.984326 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab.
Entry for principal kudu/127.1.8.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab.
W20260924 17:48:09.984517 23445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:35727: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
I20260924 17:48:09.985145 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:35727
--webserver_interface=127.1.8.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab
--principal=kudu/127.1.8.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:44845,127.1.8.189:37613,127.1.8.188:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:48:10.015612 23370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } attempt: 3
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:10.023197 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.016042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:49024)
0924 17:48:10.016108 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.016121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:10.016179 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:10.016329 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:10.016330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:10.016503 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:10.016568 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.016570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.017262 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.017263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.017810 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.017812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.018210 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.018210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.018294 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.018798 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:10.018803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:10.020309 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:10.022122 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.022123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.022124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.022295 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.022547 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.022548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.022548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.022606 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.022950 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:10.022951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:10.022992 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:48:10.023040 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 17:48:10.023123 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 17:48:10.023334 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.016118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.1.8.189:37613)
0924 17:48:10.016196 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.016198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.016208 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:48:10.016237 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.016238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.016254 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.016323 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.016674 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.017175 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.017907 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.018130 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.018213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.020402 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.020404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.020404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.020414 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.022030 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.022370 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.022371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.022372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.022456 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.022673 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.022675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.022675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.022856 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.022966 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.023088 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.023194 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:48:10.023811 23370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:10.024320 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.024193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38564)
0924 17:48:10.024253 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.024290 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 17:48:10.024650 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.024558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38574)
0924 17:48:10.024604 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.024632 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 17:48:10.024781 23370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:35727: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
I20260924 17:48:10.033756 23445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } attempt: 1
I20260924 17:48:10.034228 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.034098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38590)
0924 17:48:10.034159 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.034197 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:48:10.034595 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.034498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38598)
0924 17:48:10.034544 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.034572 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:48:10.034680 23445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:35727: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
I20260924 17:48:10.055987 23370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } attempt: 1
I20260924 17:48:10.056523 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.056318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38610)
0924 17:48:10.056403 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.056473 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 17:48:10.057032 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.056920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38620)
0924 17:48:10.056972 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.057006 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 17:48:10.057185 23370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:35727: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
W20260924 17:48:10.071444 23452 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:48:10.071668 23452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:10.071686 23452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:10.071708 23452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:10.072881 23452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:10.072926 23452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:10.072940 23452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:10.072953 23452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:10.072965 23452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:10.074477 23452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:44845,127.1.8.189:37613,127.1.8.188:35727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab
--principal=kudu/127.1.8.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.8.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:10.074682 23452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:10.074858 23452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:10.079774 23458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:10.079963 23457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:10.079811 23460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:10.079830 23452 server_base.cc:1034] running on GCE node
I20260924 17:48:10.080334 23452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:10.080590 23452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:10.081756 23452 hybrid_clock.cc:648] HybridClock initialized: now 1790272090081742 us; error 43 us; skew 500 ppm
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:10.084245 23452 init.cc:377] Logged in from keytab as kudu/127.1.8.188@KRBTEST.COM (short username kudu)
I20260924 17:48:10.084751 23452 webserver.cc:460] Webserver started at http://127.1.8.188:44973/ using document root <none> and password file <none>
I20260924 17:48:10.084980 23452 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:10.085048 23452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:10.085201 23452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:10.086310 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data/instance:
uuid: "25be96834f314928b622c55b7b6946c8"
format_stamp: "Formatted at 2026-09-24 17:48:10 on dist-test-slave-r7hf"
server_key: "ef7541b81e47ad7ba2022170b88d4f96"
server_key_iv: "19da59dbf3d5efb0429fd9377f12b1c2"
server_key_version: "encryptionkey@0"
I20260924 17:48:10.086737 23452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal/instance:
uuid: "25be96834f314928b622c55b7b6946c8"
format_stamp: "Formatted at 2026-09-24 17:48:10 on dist-test-slave-r7hf"
server_key: "ef7541b81e47ad7ba2022170b88d4f96"
server_key_iv: "19da59dbf3d5efb0429fd9377f12b1c2"
server_key_version: "encryptionkey@0"
I20260924 17:48:10.088071 23452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:48:10.088809 23467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:10.089013 23452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:48:10.089113 23452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal
uuid: "25be96834f314928b622c55b7b6946c8"
format_stamp: "Formatted at 2026-09-24 17:48:10 on dist-test-slave-r7hf"
server_key: "ef7541b81e47ad7ba2022170b88d4f96"
server_key_iv: "19da59dbf3d5efb0429fd9377f12b1c2"
server_key_version: "encryptionkey@0"
I20260924 17:48:10.089198 23452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:10.098860 23452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:10.100915 23452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:10.101094 23452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:10.102810 23445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } attempt: 2
I20260924 17:48:10.103233 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.103104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38630)
0924 17:48:10.103163 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.103196 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 17:48:10.103616 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.103512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38632)
0924 17:48:10.103568 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.103593 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 17:48:10.103727 23445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:35727: Network error: Client connection negotiation failed: client connection to 127.1.8.188:35727: connect: Connection refused (error 111)
I20260924 17:48:10.105551 23452 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.188:35727
I20260924 17:48:10.105579 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.188:35727 every 8 connection(s)
I20260924 17:48:10.105937 23452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
I20260924 17:48:10.107108 23520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:10.108680 23520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:10.110210 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23452
I20260924 17:48:10.110311 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790271791394214-1058-0/minicluster-data/master-2/wal/instance
I20260924 17:48:10.110563 1058 external_mini_cluster.cc:1428] Setting key c55f6b92346d875188280b5a92a765bc
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:10.118791 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.109427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.1.8.190:44845)
0924 17:48:10.109488 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.109490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.109668 (+ 178us) server_negotiation.cc:407] Connection header received
0924 17:48:10.110124 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.110125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.110137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.110223 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.111225 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.111716 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.112752 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.112958 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.113054 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.115367 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.115369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.115370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.115379 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.117208 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.117767 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.117768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.117768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.117830 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.118077 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.118078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.118078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.118228 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.118305 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.118611 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.118695 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:48:10.118984 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.109380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:44845 (local address 127.0.0.1:56766)
0924 17:48:10.109544 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.109584 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:10.110029 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:10.110303 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:10.110307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:10.110744 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:10.111112 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.111119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.111836 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.111837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.112643 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.112645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.113084 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.113086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.113202 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.113739 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:10.113750 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:10.115274 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:10.117327 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.117334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.117336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.117693 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.117906 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.117907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.117907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.118009 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.118297 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:10.118302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:10.118470 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 17:48:10.118588 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 17:48:10.118743 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260924 17:48:10.119851 23520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:10.120227 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.110941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38638 (local address 127.1.8.188:35727)
0924 17:48:10.111123 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.111128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.111153 (+ 25us) server_negotiation.cc:407] Connection header received
0924 17:48:10.111209 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.111211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.111239 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.111325 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.111929 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.112703 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.113507 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.113932 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.114009 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.116485 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.116493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.116493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.116501 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.118497 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.119199 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.119202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.119202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.119274 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.119563 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.119564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.119564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.119746 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.119903 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.119999 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.120097 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:48:10.127170 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.120161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:37613 (local address 127.0.0.1:49028)
0924 17:48:10.120253 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.120265 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:10.120314 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:10.120492 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:10.120493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:10.120642 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:10.120696 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.120698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.121512 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.121513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.122177 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.122179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.122677 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.122678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.122755 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.123133 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:10.123138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:10.124538 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:10.126091 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.126092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.126093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.126260 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.126528 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.126528 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.126529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.126591 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.126844 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:10.126844 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:10.126889 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:48:10.126968 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 17:48:10.127067 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 17:48:10.127172 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.120214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.1.8.189:37613)
0924 17:48:10.120259 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.120261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.120311 (+ 50us) server_negotiation.cc:407] Connection header received
0924 17:48:10.120378 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.120379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.120392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.120439 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.120796 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.121406 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.122256 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.122609 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.122660 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.124628 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.124629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.124629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.124638 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.125999 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.126351 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.126352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.126352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.126460 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.126656 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.126657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.126658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.126775 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.126852 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.127015 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.127090 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:48:10.127547 23520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:10.128885 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.121169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.1.8.190:44845)
0924 17:48:10.121231 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.121233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.121240 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:48:10.121260 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.121260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.121269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.121326 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.121909 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.122482 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.123402 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.123598 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.123674 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.125975 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.125976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.125977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.125985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.127769 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.128175 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.128176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.128176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.128236 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.128437 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.128438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.128438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.128554 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.128673 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.128762 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.128823 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272090, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:10.132323 23370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } attempt: 2
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:10.134686 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.127812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38640)
0924 17:48:10.127858 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.127869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:10.127906 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:10.128033 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:10.128034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:10.128169 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:10.128209 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.128210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.129082 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.129083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.129841 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.129843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.130220 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.130220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.130265 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.130690 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:10.130695 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:10.131762 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:10.133579 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.133580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.133580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.133772 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.133972 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.133973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.133973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.134050 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.134317 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:10.134317 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:10.134349 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 17:48:10.134450 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 17:48:10.134538 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 17:48:10.134764 23524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.127859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.1.8.188:35727)
0924 17:48:10.127897 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.127899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.127907 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:48:10.127957 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.127957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.127969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.128032 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.128306 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.129005 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.129915 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.130135 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.130220 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.131855 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.131856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.131856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.131862 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.133497 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.133847 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.133848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.133848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.133901 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.134128 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.134128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.134129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.134234 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.134278 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.134489 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.134597 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":18}
I20260924 17:48:10.137316 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.129752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49034 (local address 127.1.8.189:37613)
0924 17:48:10.129792 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.129794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.129803 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:48:10.129844 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.129845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.129854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.129907 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.130363 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.130856 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.131533 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.131759 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.131844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.134436 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.134437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.134437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.134445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.136046 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.136467 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.136468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.136469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.136538 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.136781 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.136782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.136782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.136910 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.137070 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.137140 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.137225 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:48:10.137687 23520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: Bootstrap starting.
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:10.138474 23520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:10.138872 23520 log.cc:824] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:10.139863 23520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: No bootstrap required, opened a new log
I20260924 17:48:10.139884 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:10.141013 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.132569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38656)
0924 17:48:10.132622 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.132633 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:10.132670 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:10.133753 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:10.133755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:10.133907 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:10.133968 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.133970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.135013 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.135023 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.135848 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.135851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.136284 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.136285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.136357 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.136728 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:10.136733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:10.138060 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:10.139924 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.139925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.139926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.140105 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.140326 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.140326 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.140327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.140385 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.140673 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:10.140675 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:10.140720 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 17:48:10.140789 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 17:48:10.140890 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 17:48:10.141024 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.132755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.1.8.188:35727)
0924 17:48:10.133615 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.133617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.133624 (+ 7us) server_negotiation.cc:407] Connection header received
0924 17:48:10.133643 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.133644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.133654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.133722 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.134067 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.134902 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.135934 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.136209 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.136270 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.138143 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.138144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.138144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.138152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.139835 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.140192 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.140194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.140194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.140251 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.140460 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.140461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.140462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.140597 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.140669 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.140825 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.140910 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":139,"threads_started":1}
I20260924 17:48:10.140900 23344 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7de764c3a1764302a5f12a159bd3a9a9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56770
I20260924 17:48:10.142096 23520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.142253 23520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25be96834f314928b622c55b7b6946c8, State: Initialized, Role: FOLLOWER
I20260924 17:48:10.142341 23520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.142565 23370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9: Bootstrap starting.
I20260924 17:48:10.142699 23520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:10.142952 23527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.143055 23527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.143288 23370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:10.143656 23370 log.cc:824] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:10.144516 23370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9: No bootstrap required, opened a new log
W20260924 17:48:10.144953 23539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:10.145056 23539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:48:10.145927 23370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.146072 23370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de764c3a1764302a5f12a159bd3a9a9, State: Initialized, Role: FOLLOWER
I20260924 17:48:10.146145 23370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.146443 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.146492 23370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:10.146714 23344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 17:48:10.146775 23344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:10.144955 23452 master_runner.cc:186] Error getting master registration for 127.1.8.190:44845: OK, instance_id { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" instance_seqno: 1790272089787374 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:48:10.147429 23452 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:48:10.147565 23344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.147799 23344 leader_election.cc:288] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [CANDIDATE]: Term 1 election: Requested vote from peers fd3793cdca1b4accbec26807d6650574 (127.1.8.189:37613), 25be96834f314928b622c55b7b6946c8 (127.1.8.188:35727)
I20260924 17:48:10.147820 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.148382 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7de764c3a1764302a5f12a159bd3a9a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25be96834f314928b622c55b7b6946c8"
I20260924 17:48:10.148479 23494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [term 0 FOLLOWER]: Advancing to term 1
W20260924 17:48:10.148892 23552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:10.148938 23552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:48:10.149439 23494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de764c3a1764302a5f12a159bd3a9a9 in term 1.
I20260924 17:48:10.150928 23321 leader_election.cc:302] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25be96834f314928b622c55b7b6946c8, 7de764c3a1764302a5f12a159bd3a9a9; no voters:
I20260924 17:48:10.151162 23541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:10.151443 23541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [term 1 LEADER]: Becoming Leader. State: Replica: 7de764c3a1764302a5f12a159bd3a9a9, State: Running, Role: LEADER
I20260924 17:48:10.151683 23541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.149923 23302 master_runner.cc:186] Error getting master registration for 127.1.8.189:37613: OK, instance_id { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" instance_seqno: 1790272089953825 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:48:10.152364 23302 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:48:10.152619 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de764c3a1764302a5f12a159bd3a9a9. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.152832 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:10.153352 23560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:10.153703 23560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:10.155836 23494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [term 1 FOLLOWER]: Refusing update from remote peer 7de764c3a1764302a5f12a159bd3a9a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:48:10.156297 23558 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:48:10.157859 23527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de764c3a1764302a5f12a159bd3a9a9. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.157949 23527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.160611 23527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.160703 23527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.161288 23541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.161375 23541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:10.161594 23561 mvcc.cc:205] Tried to move back new op lower bound from 7332954481275867136 to 7332954481263099904. Current Snapshot: MvccSnapshot[applied={T|T < 7332954481275867136}]
I20260924 17:48:10.161849 23562 mvcc.cc:205] Tried to move back new op lower bound from 7332954481275867136 to 7332954481263099904. Current Snapshot: MvccSnapshot[applied={T|T < 7332954481275867136}]
I20260924 17:48:10.167163 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.158701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.1.8.190:44845)
0924 17:48:10.158743 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.158745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.159236 (+ 491us) server_negotiation.cc:407] Connection header received
0924 17:48:10.159297 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.159298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.159309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.159369 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.160113 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.160856 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.161917 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.162227 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.162268 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.163902 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.163904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.163904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.163912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.165807 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.166266 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.166268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.166268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.166343 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.166578 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.166579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.166579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.166716 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.166832 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.166966 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.167066 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:48:10.170275 23560 catalog_manager.cc:1347] Generated new cluster ID: 29f694280c1e44fd9f088f6dccc8fa81
I20260924 17:48:10.170331 23560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:10.175056 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.159070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.1.8.188:35727)
0924 17:48:10.159116 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.159118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.167376 (+ 8258us) server_negotiation.cc:407] Connection header received
0924 17:48:10.167695 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.167696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.167711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.167771 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.168420 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.169212 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.170242 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.170628 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.170673 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.172095 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.172096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.172097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.172104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.173749 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.174138 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.174139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.174139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.174210 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.174464 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.174465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.174466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.174629 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.174732 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.174857 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.174954 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 17:48:10.178947 23450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.158915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.1.8.189:37613)
0924 17:48:10.158961 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.158963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.159386 (+ 423us) server_negotiation.cc:407] Connection header received
0924 17:48:10.159503 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.159505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.159515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.159545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.160263 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.160952 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.167446 (+ 6494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.167775 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.167820 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.169560 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.169561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.169562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.169571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.171461 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.177972 (+ 6511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.177975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.177975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.178051 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.178275 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.178275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.178276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.178446 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.178536 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.178746 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.178837 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 17:48:10.179245 23560 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:10.180634 23560 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:10.187597 23560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9: Generated new TSK 0
I20260924 17:48:10.187825 23560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 17:48:10.194841 23445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } attempt: 3
Sep 24 17:48:10 dist-test-slave-r7hf krb5kdc[23289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272089, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:10.203761 23448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.195181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:35727 (local address 127.0.0.1:38680)
0924 17:48:10.195239 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:10.195250 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:10.195291 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:10.195440 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:10.195440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:10.195631 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:10.195685 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.195687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.196656 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.196657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.197485 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:10.197488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.198022 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.198023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:10.198080 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.198625 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:10.198631 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:10.200261 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:10.202251 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.202252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.202252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.202498 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.202717 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:10.202718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:10.202718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:10.202786 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:10.203129 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:10.203130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:10.203182 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 17:48:10.203248 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 17:48:10.203610 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:48:10.204456 23525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:10.195221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.1.8.188:35727)
0924 17:48:10.195264 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.195266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.195280 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:48:10.195348 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.195349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.195359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.195429 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.195781 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.196537 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.197586 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:10.197889 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:10.197950 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:10.200378 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:10.200379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:10.200380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:10.200388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:10.202166 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.202585 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.202587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.202587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.202651 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:10.202849 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:10.202850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:10.202851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:10.203027 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:10.203096 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:10.204229 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0924 17:48:10.204319 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:48:10.207669 23445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: Bootstrap starting.
I20260924 17:48:10.208453 23445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:10.208750 23445 log.cc:824] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:10.209690 23445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: No bootstrap required, opened a new log
I20260924 17:48:10.211946 23445 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.212109 23445 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3793cdca1b4accbec26807d6650574, State: Initialized, Role: FOLLOWER
I20260924 17:48:10.212184 23445 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } }
I20260924 17:48:10.212535 23445 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:10.212652 23418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:10.213656 23418 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [term 1 FOLLOWER]: Refusing update from remote peer 7de764c3a1764302a5f12a159bd3a9a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:48:10.214087 23568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 17:48:10.214473 23585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:10.214520 23585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:48:10.215075 23419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7de764c3a1764302a5f12a159bd3a9a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fd3793cdca1b4accbec26807d6650574"
I20260924 17:48:10.215942 23419 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7de764c3a1764302a5f12a159bd3a9a9 in term 1.
I20260924 17:48:10.219401 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.219578 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.219669 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7de764c3a1764302a5f12a159bd3a9a9. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.219764 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.219831 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.219928 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:10.223080 23588 mvcc.cc:205] Tried to move back new op lower bound from 7332954481396645888 to 7332954481263099904. Current Snapshot: MvccSnapshot[applied={T|T < 7332954481275867136}]
W20260924 17:48:10.228335 23373 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.226708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.1.8.190:44845)
0924 17:48:10.226766 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.226769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.227613 (+ 844us) server_negotiation.cc:407] Connection header received
0924 17:48:10.227719 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.227721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.227733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.227762 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.228259 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56800: BlockingRecv error: recv got EOF from 127.0.0.1:56800 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 17:48:10.229382 23525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.227324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.1.8.188:35727)
0924 17:48:10.227386 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.227388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.228919 (+ 1531us) server_negotiation.cc:407] Connection header received
0924 17:48:10.228954 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.228955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.228965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.229018 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.229315 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38696: BlockingRecv error: recv got EOF from 127.0.0.1:38696 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 17:48:10.229889 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.227096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.1.8.189:37613)
0924 17:48:10.227135 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.227136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.228725 (+ 1589us) server_negotiation.cc:407] Connection header received
0924 17:48:10.228790 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.228791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.228804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.228881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.229817 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49054: BlockingRecv error: recv got EOF from 127.0.0.1:49054 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:48:10.235668 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.234697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.1.8.189:37613)
0924 17:48:10.234741 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.234743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.235020 (+ 277us) server_negotiation.cc:407] Connection header received
0924 17:48:10.235144 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.235145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.235167 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.235201 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.235608 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49068: BlockingRecv error: recv got EOF from 127.0.0.1:49068 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:48:10.239648 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.238810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.1.8.189:37613)
0924 17:48:10.238847 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.238848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.239090 (+ 242us) server_negotiation.cc:407] Connection header received
0924 17:48:10.239186 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.239187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.239199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.239228 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.239591 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49082: BlockingRecv error: recv got EOF from 127.0.0.1:49082 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:48:10.244069 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.243097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.1.8.189:37613)
0924 17:48:10.243134 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.243136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.243441 (+ 305us) server_negotiation.cc:407] Connection header received
0924 17:48:10.243603 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.243604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.243616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.243648 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.244013 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49088: BlockingRecv error: recv got EOF from 127.0.0.1:49088 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:48:10.250666 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.249716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.1.8.189:37613)
0924 17:48:10.249757 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.249758 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.250043 (+ 285us) server_negotiation.cc:407] Connection header received
0924 17:48:10.250149 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.250150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.250162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.250193 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.250608 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49102: BlockingRecv error: recv got EOF from 127.0.0.1:49102 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 17:48:10.261106 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.260129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.1.8.189:37613)
0924 17:48:10.260178 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.260180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.260499 (+ 319us) server_negotiation.cc:407] Connection header received
0924 17:48:10.260609 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.260610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.260622 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.260652 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.261044 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49108: BlockingRecv error: recv got EOF from 127.0.0.1:49108 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 17:48:10.279740 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.278642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.1.8.189:37613)
0924 17:48:10.278692 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.278694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.279085 (+ 391us) server_negotiation.cc:407] Connection header received
0924 17:48:10.279200 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.279202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.279215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.279246 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.279679 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49110: BlockingRecv error: recv got EOF from 127.0.0.1:49110 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 17:48:10.314136 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.313345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.1.8.189:37613)
0924 17:48:10.313393 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.313395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.313571 (+ 176us) server_negotiation.cc:407] Connection header received
0924 17:48:10.313627 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.313629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.313642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.313689 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.314077 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49116: BlockingRecv error: recv got EOF from 127.0.0.1:49116 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 17:48:10.380031 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.379345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.1.8.189:37613)
0924 17:48:10.379399 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.379402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.379505 (+ 103us) server_negotiation.cc:407] Connection header received
0924 17:48:10.379539 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.379540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.379552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.379636 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.379969 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49132: BlockingRecv error: recv got EOF from 127.0.0.1:49132 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 17:48:10.509948 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.509316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.1.8.189:37613)
0924 17:48:10.509405 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.509407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.509421 (+ 14us) server_negotiation.cc:407] Connection header received
0924 17:48:10.509483 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.509484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.509494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.509558 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.509861 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49134: BlockingRecv error: recv got EOF from 127.0.0.1:49134 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 17:48:10.685271 23568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7de764c3a1764302a5f12a159bd3a9a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de764c3a1764302a5f12a159bd3a9a9" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 44845 } } peers { permanent_uuid: "fd3793cdca1b4accbec26807d6650574" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 37613 } } peers { permanent_uuid: "25be96834f314928b622c55b7b6946c8" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 35727 } } }
I20260924 17:48:10.685456 23568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7de764c3a1764302a5f12a159bd3a9a9 [sys.catalog]: This master's current role is: LEADER
W20260924 17:48:10.767968 23450 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:10.767281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.1.8.189:37613)
0924 17:48:10.767354 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:10.767356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:10.767411 (+ 55us) server_negotiation.cc:407] Connection header received
0924 17:48:10.767457 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:10.767458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:10.767468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:10.767521 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:10.767871 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49150: BlockingRecv error: recv got EOF from 127.0.0.1:49150 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 17:48:11.145519 23539 catalog_manager.cc:1259] Loaded cluster ID: 29f694280c1e44fd9f088f6dccc8fa81
I20260924 17:48:11.145628 23539 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: loading cluster ID for follower catalog manager: success
I20260924 17:48:11.146574 23539 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: acquiring CA information for follower catalog manager: success
I20260924 17:48:11.146900 23539 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25be96834f314928b622c55b7b6946c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:48:11.215016 23585 catalog_manager.cc:1259] Loaded cluster ID: 29f694280c1e44fd9f088f6dccc8fa81
I20260924 17:48:11.215098 23585 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: loading cluster ID for follower catalog manager: success
I20260924 17:48:11.215957 23585 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: acquiring CA information for follower catalog manager: success
I20260924 17:48:11.216287 23585 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd3793cdca1b4accbec26807d6650574: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:48:11.283926 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:11.281228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.1.8.189:37613)
0924 17:48:11.281376 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:11.281378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:11.281464 (+ 86us) server_negotiation.cc:407] Connection header received
0924 17:48:11.281561 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:11.281562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:11.281647 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:11.281724 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:11.281878 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.282365 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.283204 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.283394 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.283455 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:11.283591 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:11.283698 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:11.283756 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:11.283790 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 17:48:11.283807 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260924 17:48:11.288528 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:11.285870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.1.8.188:35727)
0924 17:48:11.285992 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:11.285995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:11.286004 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:48:11.286082 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:11.286083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:11.286121 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:11.286215 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:11.286327 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.286896 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.287680 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.287896 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.287984 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:11.288162 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:11.288269 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:11.288376 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:11.288400 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 17:48:11.288423 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260924 17:48:11.293571 23669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:11.290295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.1.8.190:44845)
0924 17:48:11.290425 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:11.290428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:11.290438 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:48:11.290578 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:11.290584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:11.290653 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:11.290769 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:11.290961 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.291746 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.292675 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.292961 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.293026 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:11.293197 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:11.293301 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:11.293381 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:11.293436 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 17:48:11.293458 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260924 17:48:11.293855 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:11.290766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.1.8.188:35727)
0924 17:48:11.290952 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:11.290954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:11.291073 (+ 119us) server_negotiation.cc:407] Connection header received
0924 17:48:11.291108 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:11.291109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:11.291120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:11.291180 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:11.291278 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.291999 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.293011 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.293300 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.293342 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:11.293564 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:11.293596 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:11.293626 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:11.293759 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 17:48:11.293789 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260924 17:48:11.294142 23657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:11.290465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.1.8.189:37613)
0924 17:48:11.290521 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:11.290523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:11.290627 (+ 104us) server_negotiation.cc:407] Connection header received
0924 17:48:11.290665 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:11.290666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:11.290677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:11.290761 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:11.291140 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.291869 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.293020 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:11.293308 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:11.293343 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:11.293565 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:11.293595 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:11.293626 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:11.294041 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 17:48:11.294070 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 17:48:11.295643 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23302
I20260924 17:48:11.301386 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23377
I20260924 17:48:11.306630 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23452
2026-09-24T17:48:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2815 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 17:48:11 dist-test-slave-r7hf krb5kdc[23674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 17:48:11 dist-test-slave-r7hf krb5kdc[23674](info): set up 2 sockets
Sep 24 17:48:11 dist-test-slave-r7hf krb5kdc[23674](info): commencing operation
krb5kdc: starting...
W20260924 17:48:12.370775 1058 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/test-user.keytab.
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T17:48:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T17:48:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.8.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
Entry for principal kudu/127.1.8.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab.
I20260924 17:48:12.497653 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45147
--webserver_interface=127.1.8.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45147,127.1.8.189:33511,127.1.8.188:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 17:48:12.580120 23687 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:48:12.580358 23687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:12.580375 23687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:12.580397 23687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:12.581631 23687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:12.581679 23687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:12.581694 23687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:12.581707 23687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:12.581719 23687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:12.583316 23687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45147,127.1.8.189:33511,127.1.8.188:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.190.keytab
--principal=kudu/127.1.8.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.190:45147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.8.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:12.583592 23687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:12.583837 23687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:12.589020 23695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:12.589028 23692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:12.589039 23693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:12.589181 23687 server_base.cc:1034] running on GCE node
I20260924 17:48:12.589512 23687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:12.589818 23687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:12.590965 23687 hybrid_clock.cc:648] HybridClock initialized: now 1790272092590924 us; error 46 us; skew 500 ppm
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:12.597344 23687 init.cc:377] Logged in from keytab as kudu/127.1.8.190@KRBTEST.COM (short username kudu)
I20260924 17:48:12.597833 23687 webserver.cc:460] Webserver started at http://127.1.8.190:38525/ using document root <none> and password file <none>
I20260924 17:48:12.598074 23687 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:12.598141 23687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:12.598287 23687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:12.599542 23687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data/instance:
uuid: "dcdb48377bdd4558a5eaace51e88303a"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "cfda7257b2bb217f56ecad75564e4718"
server_key_iv: "bf4addee9ec4ea94ff1c73df3c4be8fe"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.599915 23687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal/instance:
uuid: "dcdb48377bdd4558a5eaace51e88303a"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "cfda7257b2bb217f56ecad75564e4718"
server_key_iv: "bf4addee9ec4ea94ff1c73df3c4be8fe"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.601325 23687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:48:12.602159 23702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:12.602545 23687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 17:48:12.602674 23687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal
uuid: "dcdb48377bdd4558a5eaace51e88303a"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "cfda7257b2bb217f56ecad75564e4718"
server_key_iv: "bf4addee9ec4ea94ff1c73df3c4be8fe"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.602860 23687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:12.619817 23687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:12.622289 23687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:12.622496 23687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:12.626958 23687 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.190:45147
I20260924 17:48:12.626995 23754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.190:45147 every 8 connection(s)
I20260924 17:48:12.627470 23687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/data/info.pb
I20260924 17:48:12.628551 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:12.630178 23755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:12.633109 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23687
I20260924 17:48:12.633214 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-0/wal/instance
I20260924 17:48:12.633405 1058 external_mini_cluster.cc:1428] Setting key e5f0587d98910b557cc6875f7c646d32
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.8.189@KRBTEST.COM; defaulting to no policy
I20260924 17:48:12.640947 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.631008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.1.8.190:45147)
0924 17:48:12.631267 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.631274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.631469 (+ 195us) server_negotiation.cc:407] Connection header received
0924 17:48:12.632103 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.632105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.632127 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.632193 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.632717 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.634008 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.634932 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.635272 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.635398 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.637656 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.637663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.637663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.637670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.639363 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.639755 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.639759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.639759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.639837 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.640048 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.640049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.640050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.640283 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.640373 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.640559 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.640700 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260924 17:48:12.641034 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.630989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:45147 (local address 127.0.0.1:44612)
0924 17:48:12.631318 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.631363 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.631936 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.632201 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.632204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.632473 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.632637 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.632641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.634118 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.634120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.634839 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.634841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.635388 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.635389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.635450 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.636078 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.636087 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.637552 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.639450 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.639456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.639459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.639656 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.639904 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.639904 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.639905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.639986 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.640363 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.640368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.640433 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.640534 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.640903 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":213,"spinlock_wait_cycles":8064,"thread_start_us":102,"threads_started":1}
I20260924 17:48:12.641880 23755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:12.642243 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.642130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59816)
0924 17:48:12.642177 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.642220 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 17:48:12.642375 23706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
I20260924 17:48:12.642894 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.642800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59830)
0924 17:48:12.642846 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.642871 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 17:48:12.643091 23755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:33511: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab.
Entry for principal kudu/127.1.8.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab.
I20260924 17:48:12.659721 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:33511
--webserver_interface=127.1.8.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab
--principal=kudu/127.1.8.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45147,127.1.8.189:33511,127.1.8.188:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:48:12.692333 23755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } attempt: 1
I20260924 17:48:12.692950 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.692775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59842)
0924 17:48:12.692855 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.692912 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 17:48:12.693358 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.693264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59850)
0924 17:48:12.693308 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.693338 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 17:48:12.693518 23755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:33511: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
W20260924 17:48:12.749738 23762 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:48:12.749982 23762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:12.750000 23762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:12.750022 23762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:12.751263 23762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:12.751312 23762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:12.751327 23762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:12.751368 23762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:12.751400 23762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:12.752995 23762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45147,127.1.8.189:33511,127.1.8.188:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.189.keytab
--principal=kudu/127.1.8.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.189:33511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.8.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:12.753263 23762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:12.753507 23762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:12.758605 23767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:12.758633 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:12.758704 23762 server_base.cc:1034] running on GCE node
W20260924 17:48:12.758605 23768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:12.759047 23762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:12.759358 23762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:12.760522 23762 hybrid_clock.cc:648] HybridClock initialized: now 1790272092760487 us; error 65 us; skew 500 ppm
I20260924 17:48:12.761679 23755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } attempt: 2
I20260924 17:48:12.762115 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.761987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59852)
0924 17:48:12.762052 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.762085 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 17:48:12.762451 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.762327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59860)
0924 17:48:12.762368 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.762426 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 17:48:12.762543 23755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.189:33511: Network error: Client connection negotiation failed: client connection to 127.1.8.189:33511: connect: Connection refused (error 111)
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:12.763399 23762 init.cc:377] Logged in from keytab as kudu/127.1.8.189@KRBTEST.COM (short username kudu)
I20260924 17:48:12.763836 23762 webserver.cc:460] Webserver started at http://127.1.8.189:44637/ using document root <none> and password file <none>
I20260924 17:48:12.764079 23762 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:12.764127 23762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:12.764281 23762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:12.765385 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data/instance:
uuid: "e8ca4fff5f1a432da323bb267115b143"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "6c32c12aaaa61efc442d5d01bd6d56fd"
server_key_iv: "1ca1654d12b088e99d2bb323297d8920"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.765748 23762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal/instance:
uuid: "e8ca4fff5f1a432da323bb267115b143"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "6c32c12aaaa61efc442d5d01bd6d56fd"
server_key_iv: "1ca1654d12b088e99d2bb323297d8920"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.767203 23762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 17:48:12.767949 23777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:12.768110 23762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 17:48:12.768213 23762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal
uuid: "e8ca4fff5f1a432da323bb267115b143"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "6c32c12aaaa61efc442d5d01bd6d56fd"
server_key_iv: "1ca1654d12b088e99d2bb323297d8920"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.768285 23762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:12.786757 23762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:12.789258 23762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:12.789430 23762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:12.793633 23762 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.189:33511
I20260924 17:48:12.793669 23829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.189:33511 every 8 connection(s)
I20260924 17:48:12.794119 23762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/data/info.pb
I20260924 17:48:12.795228 23830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:12.795274 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23762
I20260924 17:48:12.795365 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-1/wal/instance
I20260924 17:48:12.795583 1058 external_mini_cluster.cc:1428] Setting key 4618eb00808c34d66e07772b97477cd7
I20260924 17:48:12.797559 23830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.8.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.8.188@KRBTEST.COM" created.
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:48:12.810470 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.798295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.1.8.190:45147)
0924 17:48:12.798358 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.798360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.798603 (+ 243us) server_negotiation.cc:407] Connection header received
0924 17:48:12.799084 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.799086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.799099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.799179 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.799801 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.800658 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.801712 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.802064 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.802142 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.807105 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.807106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.807106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.807115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.808874 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.809320 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.809322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.809322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.809392 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.809645 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.809646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.809647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.809808 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.809885 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.810224 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.810321 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 17:48:12.810592 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.798283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:45147 (local address 127.0.0.1:44622)
0924 17:48:12.798471 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.798513 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.798987 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.799249 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.799252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.799505 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.799703 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.799707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.800767 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.800767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.801622 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.801624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.802171 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.802171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.802278 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.802861 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.802869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.806997 (+ 4128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.808975 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.808983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.808985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.809238 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.809471 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.809472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.809472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.809583 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.809877 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.809881 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.810069 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.810174 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.810309 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260924 17:48:12.811236 23830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:12.820298 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.811537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59876)
0924 17:48:12.811739 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.811752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.811810 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.811946 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.811947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.812112 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.812179 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.812181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.813045 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.813046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.813931 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.813934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.814425 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.814426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.814516 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.814901 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.814910 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.817433 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.819347 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.819348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.819348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.819509 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.819695 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.819696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.819696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.819746 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.819992 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.819993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.820039 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.820119 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.820212 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260924 17:48:12.820399 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.811600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59876 (local address 127.1.8.189:33511)
0924 17:48:12.811735 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.811739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.811801 (+ 62us) server_negotiation.cc:407] Connection header received
0924 17:48:12.811870 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.811872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.811882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.811941 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.812249 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.812960 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.814022 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.814325 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.814375 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.817510 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.817520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.817520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.817527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.819255 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.819579 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.819582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.819582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.819637 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.819784 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.819785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.819785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.819921 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.819986 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.820142 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.820260 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 17:48:12.821014 23830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:12.821372 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.821266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54276)
0924 17:48:12.821309 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.821353 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 17:48:12.821517 23780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
I20260924 17:48:12.821899 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.821821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54278)
0924 17:48:12.821859 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.821883 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 17:48:12.822026 23830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:33679: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.8.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab.
Entry for principal kudu/127.1.8.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab.
I20260924 17:48:12.824707 1058 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskkSMFuh/build/release/bin/kudu
/tmp/dist-test-taskkSMFuh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:33679
--webserver_interface=127.1.8.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab
--principal=kudu/127.1.8.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.8.148:40627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.8.190:45147,127.1.8.189:33511,127.1.8.188:33679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 17:48:12.853713 23755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } attempt: 3
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:12.861129 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.854244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.1.8.189:33511)
0924 17:48:12.854315 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.854317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.854329 (+ 12us) server_negotiation.cc:407] Connection header received
0924 17:48:12.854429 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.854431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.854448 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.854512 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.854830 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.855327 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.856163 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.856383 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.856458 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.858573 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.858574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.858574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.858582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.860051 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.860447 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.860447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.860448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.860501 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.860685 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.860686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.860686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.860790 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.860839 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.860993 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.861059 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 17:48:12.861179 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.854180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59884)
0924 17:48:12.854242 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.854280 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.854336 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.854516 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.854517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.854682 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.854745 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.854747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.855413 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.855415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.856072 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.856075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.856463 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.856464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.856545 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.857002 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.857012 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.858484 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.860137 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.860138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.860139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.860359 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.860562 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.860563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.860563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.860629 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.860846 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.860846 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.860900 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.860970 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.861073 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 17:48:12.861730 23755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:12.862133 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.861997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54280)
0924 17:48:12.862055 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.862096 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:48:12.862520 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.862424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54296)
0924 17:48:12.862457 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.862489 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 17:48:12.862619 23755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:33679: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
I20260924 17:48:12.871268 23830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } attempt: 1
I20260924 17:48:12.871737 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.871607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54306)
0924 17:48:12.871665 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.871706 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 17:48:12.872134 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.872021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54316)
0924 17:48:12.872069 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.872115 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 17:48:12.872224 23830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:33679: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
I20260924 17:48:12.893782 23755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } attempt: 1
I20260924 17:48:12.894299 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.894156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54332)
0924 17:48:12.894217 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.894263 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 17:48:12.894619 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.894542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54348)
0924 17:48:12.894569 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.894597 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 17:48:12.894732 23755 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.8.188:33679: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
W20260924 17:48:12.908833 23837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 17:48:12.909080 23837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 17:48:12.909101 23837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 17:48:12.909184 23837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 17:48:12.910547 23837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 17:48:12.910601 23837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 17:48:12.910619 23837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 17:48:12.910661 23837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 17:48:12.910692 23837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 17:48:12.912372 23837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.8.148:40627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.8.190:45147,127.1.8.189:33511,127.1.8.188:33679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/krb5kdc/kudu_127.1.8.188.keytab
--principal=kudu/127.1.8.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.8.188:33679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.8.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 16:40:09 UTC on ccycloud.quasar-meryvx.root.comops.site
build id 4670649
I20260924 17:48:12.912648 23837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 17:48:12.912930 23837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 17:48:12.917984 23843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:12.917990 23845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 17:48:12.918099 23842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 17:48:12.918448 23837 server_base.cc:1034] running on GCE node
I20260924 17:48:12.918686 23837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 17:48:12.919008 23837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 17:48:12.920171 23837 hybrid_clock.cc:648] HybridClock initialized: now 1790272092920151 us; error 37 us; skew 500 ppm
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 17:48:12.922652 23837 init.cc:377] Logged in from keytab as kudu/127.1.8.188@KRBTEST.COM (short username kudu)
I20260924 17:48:12.923117 23837 webserver.cc:460] Webserver started at http://127.1.8.188:34371/ using document root <none> and password file <none>
I20260924 17:48:12.923347 23837 fs_manager.cc:359] Metadata directory not provided
I20260924 17:48:12.923413 23837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 17:48:12.923559 23837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 17:48:12.924639 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data/instance:
uuid: "671fb50a20d641e791be0014aa0cef86"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "48950d3855d93681bffb2c2d795711b3"
server_key_iv: "1356b388ecf4c458df6c6529ce409edb"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.925024 23837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal/instance:
uuid: "671fb50a20d641e791be0014aa0cef86"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "48950d3855d93681bffb2c2d795711b3"
server_key_iv: "1356b388ecf4c458df6c6529ce409edb"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.926362 23837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 17:48:12.927138 23852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 17:48:12.927305 23837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 17:48:12.927402 23837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data,/tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal
uuid: "671fb50a20d641e791be0014aa0cef86"
format_stamp: "Formatted at 2026-09-24 17:48:12 on dist-test-slave-r7hf"
server_key: "48950d3855d93681bffb2c2d795711b3"
server_key_iv: "1356b388ecf4c458df6c6529ce409edb"
server_key_version: "encryptionkey@0"
I20260924 17:48:12.927484 23837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 17:48:12.934361 23837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 17:48:12.936721 23837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 17:48:12.936887 23837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 17:48:12.940376 23830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } attempt: 2
I20260924 17:48:12.940831 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.940696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54360)
0924 17:48:12.940755 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.940797 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.8.188:33679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 17:48:12.941162 23837 rpc_server.cc:307] RPC server started. Bound to: 127.1.8.188:33679
I20260924 17:48:12.941215 23904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.8.188:33679 every 8 connection(s)
I20260924 17:48:12.941588 23837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/data/info.pb
I20260924 17:48:12.942348 23905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 17:48:12.943918 23905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.189@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
I20260924 17:48:12.949601 1058 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskkSMFuh/build/release/bin/kudu as pid 23837
I20260924 17:48:12.949710 1058 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskkSMFuh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790271791394214-1058-0/minicluster-data/master-2/wal/instance
I20260924 17:48:12.949939 1058 external_mini_cluster.cc:1428] Setting key 62bf27127ff31cab95d10607537d3b99
I20260924 17:48:12.951126 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.941132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54372)
0924 17:48:12.941179 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.941189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.941234 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.942927 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.942928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.943130 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.943173 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.943176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.944316 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.944318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.945080 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.945083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.945780 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.945780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.945828 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.946310 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.946315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.947789 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.949646 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.949647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.949647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.949885 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.950152 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.950153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.950153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.950220 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.950720 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.950721 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.950774 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.950863 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.951004 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 17:48:12.951375 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.941505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.1.8.188:33679)
0924 17:48:12.942172 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.942180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.942207 (+ 27us) server_negotiation.cc:407] Connection header received
0924 17:48:12.942732 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.942740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.942791 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.942950 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.943273 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.944231 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.945158 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.945694 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.945809 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.947858 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.947864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.947864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.947875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.949538 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.949962 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.949966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.949966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.950073 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.950292 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.950293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.950293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.950651 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.950741 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.950962 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.951112 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":556,"threads_started":1}
I20260924 17:48:12.953209 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.944589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.1.8.190:45147)
0924 17:48:12.944650 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.944652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.944800 (+ 148us) server_negotiation.cc:407] Connection header received
0924 17:48:12.944861 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.944862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.944871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.944932 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.945425 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.946040 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.946929 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.947234 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.947311 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.949842 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.949844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.949844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.949853 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.951533 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.952276 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.952277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.952278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.952353 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.952562 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.952564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.952564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.952698 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.952766 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.953032 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.953117 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 17:48:12.953217 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.944539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.190:45147 (local address 127.0.0.1:44626)
0924 17:48:12.944694 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.944736 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.944804 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.944955 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.944958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.945286 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.945350 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.945352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.946127 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.946128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.946844 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.946847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.947333 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.947334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.947385 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.948139 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.948142 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.949761 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.951601 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.951605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.951607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.952178 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.952423 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.952423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.952424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.952490 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.952769 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.952776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.952842 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.952974 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.953103 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260924 17:48:12.953522 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: Bootstrap starting.
I20260924 17:48:12.954353 23830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:12.954686 23830 log.cc:824] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:12.955473 23905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } has no permanent_uuid. Determining permanent_uuid...
I20260924 17:48:12.955600 23830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: No bootstrap required, opened a new log
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:12.957582 23830 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.957731 23830 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8ca4fff5f1a432da323bb267115b143, State: Initialized, Role: FOLLOWER
I20260924 17:48:12.957820 23830 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.958179 23830 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:12.958155 23912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:12.958307 23912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:48:12.960253 23923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:12.960306 23923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:12.961683 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.950361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.1.8.188:33679)
0924 17:48:12.950554 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.950555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.950619 (+ 64us) server_negotiation.cc:407] Connection header received
0924 17:48:12.950917 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.950918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.950928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.950995 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.952082 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.952869 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.954080 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.954292 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.954362 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.957172 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.957173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.957174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.957181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.960157 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.960589 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.960590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.960590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.960648 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.960852 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.960853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.960853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.961005 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.961198 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.961433 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.961530 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":135,"threads_started":1}
I20260924 17:48:12.964115 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.955939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.189:33511 (local address 127.0.0.1:59890)
0924 17:48:12.955981 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.955991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.956025 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.956143 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.956144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.956301 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.956356 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.956358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.957239 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.957240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.958445 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.958449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.958979 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.958981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.959060 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.959455 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.959461 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.960931 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.962963 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.962964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.962964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.963188 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.963396 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.963397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.963397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.963456 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.963809 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.963810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.963851 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.963914 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.964019 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 17:48:12.960556 23762 master_runner.cc:186] Error getting master registration for 127.1.8.190:45147: OK, instance_id { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" instance_seqno: 1790272092625610 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 17:48:12.964247 23762 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:48:12.964381 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.955992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.1.8.189:33511)
0924 17:48:12.956036 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.956039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.956047 (+ 8us) server_negotiation.cc:407] Connection header received
0924 17:48:12.956074 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.956075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.956085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.956142 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.956440 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.957160 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.958552 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.958854 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.958922 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.961041 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.961043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.961043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.961051 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.962861 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.963264 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.963266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.963266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.963325 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.963529 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.963530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.963530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.963710 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.963779 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.964223 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.964299 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:48:12.964973 23905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.190@KRBTEST.COM
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.188@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:12.969864 23755 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } attempt: 2
I20260924 17:48:12.972368 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.962897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44638 (local address 127.1.8.190:45147)
0924 17:48:12.962939 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.962942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.963080 (+ 138us) server_negotiation.cc:407] Connection header received
0924 17:48:12.963106 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.963107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.963119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.963165 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.963812 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.964521 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.965433 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.965721 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.965777 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.968535 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.968536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.968536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.968545 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.970340 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.970813 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.970814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.970814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.970872 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.971145 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.971145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.971146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.971276 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.971576 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.972192 (+ 616us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.972273 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 17:48:12.972373 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.965229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54384)
0924 17:48:12.965290 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.965299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.965340 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.965466 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.965467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.965609 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.965642 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.965643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.966580 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.966581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.967409 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.967412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.967795 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.967796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.967843 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.968250 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.968254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.969544 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.971436 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.971437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.971438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.971626 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.971784 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.971785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.971785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.971827 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.972057 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.972058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.972130 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.972189 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.972276 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 17:48:12.973097 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.965258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.1.8.188:33679)
0924 17:48:12.965295 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.965297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.965327 (+ 30us) server_negotiation.cc:407] Connection header received
0924 17:48:12.965387 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.965388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.965398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.965461 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.965721 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.966480 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.967515 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.967728 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.967800 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.969627 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.969628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.969628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.969636 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.971359 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.971689 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.971689 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.971690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.971732 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.971875 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.971875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.971876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.971984 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.972037 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.972915 (+ 878us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.973011 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 17:48:12.975000 23905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: Bootstrap starting.
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.8.190@KRBTEST.COM for kudu/127.1.8.188@KRBTEST.COM
I20260924 17:48:12.975731 23905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:12.976099 23905 log.cc:824] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:12.978535 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.970384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.1.8.188:33679)
0924 17:48:12.970706 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.970708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.970717 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:48:12.970737 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.970738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.970749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.970807 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.971139 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.971936 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.972922 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.973234 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.973297 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.975584 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.975585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.975586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.975593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.977273 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.977682 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.977683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.977684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.977746 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.977971 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.977971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.977972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.978101 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.978179 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.978340 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.978444 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260924 17:48:12.978543 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.970174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.8.188:33679 (local address 127.0.0.1:54390)
0924 17:48:12.970232 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 17:48:12.970243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 17:48:12.970277 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 17:48:12.970855 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 17:48:12.970857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 17:48:12.971005 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 17:48:12.971052 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.971054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.972023 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.972024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.972832 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 17:48:12.972835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.973503 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.973504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 17:48:12.973564 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.974036 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 17:48:12.974042 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 17:48:12.975493 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 17:48:12.977362 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.977363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.977364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.977594 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.977823 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 17:48:12.977824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 17:48:12.977824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 17:48:12.977896 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 17:48:12.978176 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 17:48:12.978177 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 17:48:12.978227 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 17:48:12.978295 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 17:48:12.978444 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
Sep 24 17:48:12 dist-test-slave-r7hf krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790272092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.8.189@KRBTEST.COM
I20260924 17:48:12.980581 23755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a: Bootstrap starting.
I20260924 17:48:12.981370 23755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a: Neither blocks nor log segments found. Creating new log.
I20260924 17:48:12.981768 23755 log.cc:824] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a: Log is configured to *not* fsync() on all Append() calls
I20260924 17:48:12.982645 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.973275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.1.8.189:33511)
0924 17:48:12.973315 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.973317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.973425 (+ 108us) server_negotiation.cc:407] Connection header received
0924 17:48:12.973587 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.973588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.973600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.973659 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.974357 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.975146 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.976320 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.976622 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:12.976668 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:12.979398 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:12.979400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:12.979400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:12.979409 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:12.981189 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.981626 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.981627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.981628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.981695 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:12.981943 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:12.981944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:12.981944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:12.982111 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:12.982318 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:12.982359 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 17:48:12.982529 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 17:48:12.983036 23755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a: No bootstrap required, opened a new log
I20260924 17:48:12.983047 23905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: No bootstrap required, opened a new log
I20260924 17:48:12.984570 23755 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.984720 23755 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdb48377bdd4558a5eaace51e88303a, State: Initialized, Role: FOLLOWER
I20260924 17:48:12.984812 23755 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.985092 23927 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:12.985164 23755 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:12.985122 23905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.985280 23905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671fb50a20d641e791be0014aa0cef86, State: Initialized, Role: FOLLOWER
I20260924 17:48:12.985252 23927 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:12.985387 23905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.985683 23905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 17:48:12.985644 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:12.985841 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 17:48:12.987296 23944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:12.987401 23944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:48:12.988835 1058 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 17:48:12.989856 23729 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dcdb48377bdd4558a5eaace51e88303a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44638
I20260924 17:48:12.989979 23729 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 17:48:12.990021 23729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:12.991137 23729 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.991357 23729 leader_election.cc:288] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [CANDIDATE]: Term 1 election: Requested vote from peers e8ca4fff5f1a432da323bb267115b143 (127.1.8.189:33511), 671fb50a20d641e791be0014aa0cef86 (127.1.8.188:33679)
I20260924 17:48:12.991701 23879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcdb48377bdd4558a5eaace51e88303a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "671fb50a20d641e791be0014aa0cef86"
I20260924 17:48:12.991832 23879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 17:48:12.992749 23879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdb48377bdd4558a5eaace51e88303a in term 1.
I20260924 17:48:12.992977 23705 leader_election.cc:302] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671fb50a20d641e791be0014aa0cef86, dcdb48377bdd4558a5eaace51e88303a; no voters:
I20260924 17:48:12.993139 23927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 17:48:12.993317 23927 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [term 1 LEADER]: Becoming Leader. State: Replica: dcdb48377bdd4558a5eaace51e88303a, State: Running, Role: LEADER
I20260924 17:48:12.993410 23927 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } }
I20260924 17:48:12.994772 23837 master_runner.cc:263] No leader master found from master 671fb50a20d641e791be0014aa0cef86
I20260924 17:48:12.994717 23804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dcdb48377bdd4558a5eaace51e88303a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e8ca4fff5f1a432da323bb267115b143"
I20260924 17:48:12.994822 23837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:48:12.994865 23804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [term 0 FOLLOWER]: Advancing to term 1
W20260924 17:48:12.995528 23951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 17:48:12.995573 23951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 17:48:12.996217 23804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcdb48377bdd4558a5eaace51e88303a in term 1.
I20260924 17:48:12.997020 23687 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260924 17:48:12.997100 23687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 17:48:12.998129 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdb48377bdd4558a5eaace51e88303a. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:12.998298 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:12.998595 23961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 17:48:12.998824 23961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 17:48:13.000514 23879 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [term 1 FOLLOWER]: Refusing update from remote peer dcdb48377bdd4558a5eaace51e88303a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:48:13.000905 23952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [LEADER]: Connected to new peer: Peer: permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:48:13.002359 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdb48377bdd4558a5eaace51e88303a. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:13.002496 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:13.003021 23804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [term 1 FOLLOWER]: Refusing update from remote peer dcdb48377bdd4558a5eaace51e88303a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 17:48:13.003355 23952 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 17:48:13.004724 23912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdb48377bdd4558a5eaace51e88303a. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:13.004815 23912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:13.006153 23835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.997577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.1.8.189:33511)
0924 17:48:12.997656 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.997658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:12.998001 (+ 343us) server_negotiation.cc:407] Connection header received
0924 17:48:12.998106 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:12.998107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:12.998119 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:12.998149 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:12.999025 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:12.999747 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:13.000817 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:13.001125 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:13.001168 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:13.003094 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:13.003096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:13.003096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:13.003104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:13.004869 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:13.005277 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:13.005279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:13.005279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:13.005333 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:13.005530 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:13.005531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:13.005531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:13.005661 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:13.005734 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:13.005971 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 17:48:13.006060 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 17:48:13.006186 23952 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:13.006283 23952 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:13.007161 23912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:13.007239 23912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:13.007637 23962 mvcc.cc:205] Tried to move back new op lower bound from 7332954492927287296 to 7332954492901634048. Current Snapshot: MvccSnapshot[applied={T|T < 7332954492927287296}]
I20260924 17:48:13.008046 23966 mvcc.cc:205] Tried to move back new op lower bound from 7332954492927287296 to 7332954492901634048. Current Snapshot: MvccSnapshot[applied={T|T < 7332954492927287296}]
I20260924 17:48:13.008929 23927 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:13.009040 23927 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a [sys.catalog]: This master's current role is: LEADER
I20260924 17:48:13.009454 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dcdb48377bdd4558a5eaace51e88303a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcdb48377bdd4558a5eaace51e88303a" member_type: VOTER last_known_addr { host: "127.1.8.190" port: 45147 } } peers { permanent_uuid: "e8ca4fff5f1a432da323bb267115b143" member_type: VOTER last_known_addr { host: "127.1.8.189" port: 33511 } } peers { permanent_uuid: "671fb50a20d641e791be0014aa0cef86" member_type: VOTER last_known_addr { host: "127.1.8.188" port: 33679 } } }
I20260924 17:48:13.009542 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 17:48:13.010247 23963 mvcc.cc:205] Tried to move back new op lower bound from 7332954492927287296 to 7332954492901634048. Current Snapshot: MvccSnapshot[applied={T|T < 7332954492927287296}]
I20260924 17:48:13.010881 23961 catalog_manager.cc:1347] Generated new cluster ID: a31ba51abdc94bb9bd6243b4a6949bea
I20260924 17:48:13.010926 23961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 17:48:13.014232 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:12.997689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.1.8.188:33679)
0924 17:48:12.997728 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:12.997730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.006566 (+ 8836us) server_negotiation.cc:407] Connection header received
0924 17:48:13.006667 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.006668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.006680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.006753 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.007434 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:13.008193 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:13.009197 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:13.009497 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:13.009539 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:13.011125 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:13.011126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:13.011127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:13.011134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:13.012911 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:13.013329 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:13.013330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:13.013330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:13.013389 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:13.013598 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:13.013599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:13.013599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:13.013729 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:13.013812 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:13.014052 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 17:48:13.014135 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 17:48:13.022011 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:13.002630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44652 (local address 127.1.8.190:45147)
0924 17:48:13.002683 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.002686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.014577 (+ 11891us) server_negotiation.cc:407] Connection header received
0924 17:48:13.014686 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.014687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.014699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.014731 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.015384 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:13.016128 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:13.017121 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:13.017421 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:13.017465 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:13.019050 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 17:48:13.019052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 17:48:13.019052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 17:48:13.019059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 17:48:13.020736 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:13.021143 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:13.021144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:13.021144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:13.021198 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 17:48:13.021388 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 17:48:13.021389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 17:48:13.021389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 17:48:13.021524 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 17:48:13.021600 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:13.021810 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 17:48:13.021909 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 17:48:13.043459 23961 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 17:48:13.044368 23961 catalog_manager.cc:1504] Loading token signing keys...
I20260924 17:48:13.056738 23961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcdb48377bdd4558a5eaace51e88303a: Generated new TSK 0
I20260924 17:48:13.057075 23961 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 17:48:13.083132 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.081258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.1.8.189:33511)
0924 17:48:13.081309 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.081311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.082355 (+ 1044us) server_negotiation.cc:407] Connection header received
0924 17:48:13.082507 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.082509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.082521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.082554 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.083053 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59918: BlockingRecv error: recv got EOF from 127.0.0.1:59918 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 17:48:13.083966 23907 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.081726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.1.8.188:33679)
0924 17:48:13.081767 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.081770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.083395 (+ 1625us) server_negotiation.cc:407] Connection header received
0924 17:48:13.083500 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.083501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.083513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.083544 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.083886 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54404: BlockingRecv error: recv got EOF from 127.0.0.1:54404 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:48:13.084769 23758 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.082075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.1.8.190:45147)
0924 17:48:13.082115 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.082116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.084178 (+ 2062us) server_negotiation.cc:407] Connection header received
0924 17:48:13.084279 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.084280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.084291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.084319 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.084707 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44664: BlockingRecv error: recv got EOF from 127.0.0.1:44664 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 17:48:13.087466 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.086555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.1.8.189:33511)
0924 17:48:13.086596 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.086597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.086861 (+ 264us) server_negotiation.cc:407] Connection header received
0924 17:48:13.087012 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.087013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.087032 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.087063 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.087415 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59924: BlockingRecv error: recv got EOF from 127.0.0.1:59924 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 17:48:13.090608 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.089703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.1.8.189:33511)
0924 17:48:13.089740 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.089742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.090000 (+ 258us) server_negotiation.cc:407] Connection header received
0924 17:48:13.090101 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.090102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.090114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.090144 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.090555 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59938: BlockingRecv error: recv got EOF from 127.0.0.1:59938 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 17:48:13.094692 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.093735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.1.8.189:33511)
0924 17:48:13.093774 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.093776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.094041 (+ 265us) server_negotiation.cc:407] Connection header received
0924 17:48:13.094143 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.094144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.094155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.094186 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.094640 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59950: BlockingRecv error: recv got EOF from 127.0.0.1:59950 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 17:48:13.100876 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.099955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.1.8.189:33511)
0924 17:48:13.099999 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.100000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.100277 (+ 277us) server_negotiation.cc:407] Connection header received
0924 17:48:13.100402 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.100403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.100415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.100446 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.100820 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59954: BlockingRecv error: recv got EOF from 127.0.0.1:59954 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 17:48:13.111162 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.110135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.1.8.189:33511)
0924 17:48:13.110182 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.110184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.110540 (+ 356us) server_negotiation.cc:407] Connection header received
0924 17:48:13.110652 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.110653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.110665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.110696 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.111102 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59956: BlockingRecv error: recv got EOF from 127.0.0.1:59956 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 17:48:13.129587 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.128537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.1.8.189:33511)
0924 17:48:13.128586 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.128588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.128912 (+ 324us) server_negotiation.cc:407] Connection header received
0924 17:48:13.129029 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.129030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.129043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.129075 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.129518 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59960: BlockingRecv error: recv got EOF from 127.0.0.1:59960 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 17:48:13.163746 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.162966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.1.8.189:33511)
0924 17:48:13.163022 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.163024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.163167 (+ 143us) server_negotiation.cc:407] Connection header received
0924 17:48:13.163246 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.163247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.163259 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.163329 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.163685 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59964: BlockingRecv error: recv got EOF from 127.0.0.1:59964 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 17:48:13.229504 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.228865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.1.8.189:33511)
0924 17:48:13.228981 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.228983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.229006 (+ 23us) server_negotiation.cc:407] Connection header received
0924 17:48:13.229077 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.229078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.229090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.229159 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.229455 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59972: BlockingRecv error: recv got EOF from 127.0.0.1:59972 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 17:48:13.359630 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.358855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.1.8.189:33511)
0924 17:48:13.358916 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.358918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.359041 (+ 123us) server_negotiation.cc:407] Connection header received
0924 17:48:13.359119 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.359120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.359133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.359199 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.359578 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59986: BlockingRecv error: recv got EOF from 127.0.0.1:59986 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 17:48:13.617529 23835 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 17:48:13.616921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.1.8.189:33511)
0924 17:48:13.616989 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:13.616991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:13.617000 (+ 9us) server_negotiation.cc:407] Connection header received
0924 17:48:13.617025 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:13.617027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:13.617036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:13.617086 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 17:48:13.617446 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59998: BlockingRecv error: recv got EOF from 127.0.0.1:59998 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 17:48:13.960726 23923 catalog_manager.cc:1259] Loaded cluster ID: a31ba51abdc94bb9bd6243b4a6949bea
I20260924 17:48:13.960800 23923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: loading cluster ID for follower catalog manager: success
I20260924 17:48:13.961609 23923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: acquiring CA information for follower catalog manager: success
I20260924 17:48:13.961869 23923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8ca4fff5f1a432da323bb267115b143: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:48:13.996026 23951 catalog_manager.cc:1259] Loaded cluster ID: a31ba51abdc94bb9bd6243b4a6949bea
I20260924 17:48:13.996101 23951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: loading cluster ID for follower catalog manager: success
I20260924 17:48:13.996874 23951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: acquiring CA information for follower catalog manager: success
I20260924 17:48:13.997136 23951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 671fb50a20d641e791be0014aa0cef86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 17:48:14.133591 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:14.130744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.1.8.189:33511)
0924 17:48:14.130882 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:14.130885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:14.130979 (+ 94us) server_negotiation.cc:407] Connection header received
0924 17:48:14.131062 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:14.131063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:14.131133 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:14.131223 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:14.131375 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.131934 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.132742 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.132955 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.133032 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:14.133226 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:14.133341 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:14.133431 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:14.133462 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 17:48:14.133484 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260924 17:48:14.137775 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:14.135482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.1.8.188:33679)
0924 17:48:14.135598 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:14.135600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:14.135610 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:48:14.135677 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:14.135678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:14.135745 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:14.135819 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:14.135923 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.136408 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.137120 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.137306 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.137355 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:14.137492 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:14.137599 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:14.137657 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:14.137688 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 17:48:14.137706 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260924 17:48:14.142458 24041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:14.139578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.1.8.188:33679)
0924 17:48:14.139638 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:14.139640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:14.139815 (+ 175us) server_negotiation.cc:407] Connection header received
0924 17:48:14.139937 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:14.139938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:14.139948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:14.139995 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:14.140124 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.140582 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.141650 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.141950 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.142009 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:14.142182 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:14.142215 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:14.142261 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:14.142309 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 17:48:14.142332 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 17:48:14.142558 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:14.139584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.1.8.189:33511)
0924 17:48:14.139674 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:14.139677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:14.139833 (+ 156us) server_negotiation.cc:407] Connection header received
0924 17:48:14.139871 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:14.139873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:14.139886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:14.139956 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:14.140058 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.140823 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.141808 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.142052 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.142117 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:14.142239 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:14.142281 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:14.142329 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:14.142446 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 17:48:14.142476 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 17:48:14.142751 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 17:48:14.139533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.1.8.190:45147)
0924 17:48:14.139729 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 17:48:14.139731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 17:48:14.139741 (+ 10us) server_negotiation.cc:407] Connection header received
0924 17:48:14.139790 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 17:48:14.139791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 17:48:14.139847 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 17:48:14.139925 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 17:48:14.140114 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.140852 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.141621 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 17:48:14.141943 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 17:48:14.142012 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 17:48:14.142215 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 17:48:14.142331 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 17:48:14.142383 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 17:48:14.142653 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 17:48:14.142679 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":58,"threads_started":1}
I20260924 17:48:14.144410 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23687
I20260924 17:48:14.149793 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23762
I20260924 17:48:14.154897 1058 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskkSMFuh/build/release/bin/kudu with pid 23837
2026-09-24T17:48:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2848 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5663 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302768 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS